Engineering Philosophy: Quality in Software


Liam Yafuso

on •

Jun 2, 2023

Engineering Philosophy: Quality in Software

The thing that sets apart the teams that achieve their outcomes from those that end up mired in the graveyard of poor execution is their ability to have a singular focus. If you know the one thing you are optimizing for, all other things can be re-assessed. The ego melts away, and all there is is the task at hand. Nothing is sacred, and everything can be reevaluated in the name of the true goal.

In my professional career, that singular focus is the delivery of working software to users that benefit from it. I have enough notches under my belt that I envision myself to be the proverbial “Wolf” of Pulp Fiction legend within the software world. You call me in, and Sh&t. Gets. Done. 

Achieving these results is fulfilling to me. It gives me purpose and meaning. But it isn’t easy.

There are so many false premises, oxymorons, and red herrings in the world of software development such that avoiding getting pulled into one of a thousand pits of poor execution is like walking on the edge of a knife.

Everywhere you look there are experts with answers in search of problems, trying to pour advice into your particular set of difficulties. Desperately hoping to feel validated if their solutions happen to work for you, and moving along to the next ear that will listen if they don’t.I believe that the difference between hand-wavy “advice” and hard-earned lessons is their closeness to the source. I hope that my words reach you as the lessons that I know they are, as they have all been hewn from the stone of real life problems, building real software applications that run real businesses and impact real users.

I come to you not as a manager or executive, but as a fellow practitioner. One of the hardened software developers who has done his fair share of the feature dev legwork, code commits, late night deployments, and legacy codebase spelunking to meet the deadlines, arbitrary or otherwise, imposed by the business.

We are each on our own journey as students of our craft. At some point we may become lucky enough to find those who confide in us, and look to us for wisdom and direction. We impart what we can while knowing full well we are still students ourselves. It is in that headspace of empathy and humility that I offer you the distilled learnings of my time thus far in the field of software development. I believe they will resonate with your problems and situation, for I have found these learnings to be at the core of every major problem in the industry I have ever faced.

Lesson 1 - The definition of Quality in Software.

When we all first began to write code, the difference between good code and bad code was whether it worked. We would wrestle with a problem for hours copy pasting various answers from forums on the internet in the hopes that one of them might finally get us that sweet release of functioning software. If and when we reached that solution, that magic piece of hacky, ugly, beautiful, at-least-it-freaking-works code, we would rejoice and move forward. Code reviews be damned, we had achieved success. We were junior, surrounded by more experienced and qualified professionals who would occasionally stop whatever nit-picky nerd holy war they were waging with one another to look down their noses at us, and we took whatever wins we could get.

Over time, we begin to feel more comfortable with solving common problems and implementing common features. More and more of it is muscle memory, and so we begin to have the mental capacity to consider the performance implications of our copy pasta’d implementations. Oh snap, is that an oh of en squared?? Well that’s not acceptable. Perhaps in the past that would have flown under the radar, but no more. We are junior no longer. And so we carry on with our newfound capability, subbing out arrays for hashes to get that sweet sweet oh of one lookup time because memory on modern machines is infinite right? All is right in our world, the code is perfect.

Then the business asks for what they see as a slight variation on a couple of existing features. Everywhere in the app where this happens, we also want to support that option. Simple enough right? Except that this particular feature isn’t written as a single feature. As you look through the various sources you’ve authored, you realize that you’ve implemented the logic for this feature in 30+ different instances. While it may look to users like a single type of functionality, it is in practice many different features that all just seem similar in behavior. To update all of these many disparate code blocks, it would be best to unify them into a single configurable logic engine that is reused throughout the codebase. This aggregated engine would represent all of a certain “class” of functionality, if you will. So you write your new class that can handle all of a variety of similar-seeming use cases, and bind it to your 30+ different parts of the codebase. Suddenly you can add your new feature all at once. It is beautiful. You turn your newfound mental eye towards all repeated code in the codebase, unifying everything into new logic engines where necessary, or old engines when you find you can stretch a preexisting abstraction just enough to cram a new use case into it. An untold number of refactors later you’ve done it! Not a single repeated line of code. You’ve architected the perfect, glorious abstraction for every feature in the application.

The code is absolute perfection, and through your robust implementation the business has begun to scale. Your team has grown by a handful of new members and revenue projections that were previously out of reach are now taken for granted. The business expects that by doubling the development team, they should get double the feature outputs. This would be true if it weren’t for the fact that all these new team members are peasants! They’re constantly complaining about the complexity of the codebase and every time they touch any of the aggregate classes you have to review and inevitably fix their code as they don’t understand the implications of their changes. One day there is a serious production bug. One of the new devs has changed a seemingly innocuous line of code that caused a whole suite of features to cease functioning properly. You’re up late into the night pushing hot fixes. You are now a single point of failure for the engineering team. Your teammates should be grateful to your genius! Some of them are, but most are frustratingly the opposite. You have written a masterpiece but these plebeians are too ignorant to appreciate the genius that is your application! 

Or perhaps it is not a masterpiece at all, but rather a cluttered mess of increasingly poor abstractions that have been tortured and twisted into shapes unrecognizable from when they began. Perhaps the deeply coupled nature of all the code and low unit test coverage have resulted in a minefield of undocumented assumptions that are blowing the limbs off of unsuspecting junior devs on a regular basis. It is a tragedy, but many of us get stuck at this phase of self-development. We have spent so much time getting better and better and are surrounded by devs more junior than us, so it is easy to brush off their complaints and attribute them all to a lack of expertise. And while it is so easy to project all the problems outward, this is the most critical time to redirect them inward. Look at all these complaints not to find where they are flawed, but to find where they are valid. Is the code you’ve written complicated? Is it difficult to understand? Have you built the tools and processes that would act as guard rails for new developers who are as junior now as you were in the past? The answer is as clear as it is painful. You now look back over the thousands of lines you’ve committed with a new perspective and realize what you’ve done. You’ve created a monster. A beast so fickle and picky that only its original owner has any chance of taming it, and frankly, if you’re honest with yourself, it’s gotten difficult even for you. Those late night releases you spent wrestling with it are fresh in your mind. As you review your work of the last few months and years you are humbled to the ground. You are rocked to your core, for you have learned the deepest and most pernicious lesson about quality of code.

In software, quality isn’t just about whether the product “works” or is “performant”. Quality is about how easy it is to add new features, and how effectively new team members can understand and inherit the code.

Do the abstractions you’ve introduced make sense within the domain? Is the complexity you’ve introduced through your abstractions actually justified by the problems it solves? Or have you merged groups of functionality together into massive core classes simply to remove the amount of repeated lines of code, regardless of whether those lines may need to diverge in the future? Despite our primal urges to DRY (don’t-repeat-yourself) up our code, repeated code is not itself a sin. If two pieces of repeated logic always change in tandem, then they should be unified. If two pieces of code change independently but happen right now to have the same logic, then they should not be unified.

The guiding principle of an abstraction should always be “Does this make the code easier to work with and understand?” The introduction of complexity is only ever justified if it solves for even greater complexity. 

With this in mind, automated testing becomes a requirement for software to be considered high quality. A well-maintained test harness allows even new developers to have confidence that they are not creating bugs when adding new features. Think of your codebase as an organism. A living, breathing organism that will, as a necessity for survival, need to grow and evolve to keep up with its changing environment. The features in your application are its muscles and sinew, the UI is its colorful hide, and the test suite is its nervous system. Tests are how the application can detect if it’s injured or uncomfortable. Ideally a nervous system is a network of sensors reaching throughout all the muscles and parts of the body to give continuous real-time feedback about the state of each piece. If a particular action causes pain, the organism knows to stop taking that particular action and to find another way. Without that feedback, all the muscles are still there, all the capabilities are still there, but it might run into a wall or break a limb and continue plowing forward without even knowing how much it's been damaged or the degree to which it’s limbs no longer function.

With a well-maintained suite of unit and integration tests however, we can keep track of the health of all existing features with each new evolution and iteration we bring. We can add guide rails and protective gear to all the gnarliest parts of the codebase such that even the junior folks can attempt to tackle changes to the most complex parts of the system.

Perhaps they will struggle to actually implement new functionality, but at least they won’t accidentally break anything. Over time your codebase has gone from a mindless beast that feels no pain, to a more sensitive and self-aware entity that communicates with you about its health and injuries. As a leader on your team you are creating an environment that is much more welcoming and healthy for your peers.

Setting new members up for success by lowering the barrier to entry is critical.
Individual members of any team will come and go, and the quality of the code you write will determine whether your application is resilient to changes in the team roster or not. An experienced lead developer will guide their team towards the metric of maintainability, knowing that one of the most important user personas of the system is the fellow software developer straining to grow and evolve the application to adapt to the ever-changing future.

This is the most difficult challenge around quality yet. How do you know if your code is readable to other people who know less than you?
Ask for feedback from your team. Onboard them into the system yourself and pay attention to what they find confusing and where they get stuck. Pay attention to your own cognitive load when working on various parts of the system. Are there deep dark caves in your source that make you feel like your brain is going to explode when you work with them? Each of these instances is an opportunity for you to decouple your abstractions and add more robust test harnesses. Take pride in it! For you are facing the most meaningful of challenges and taking on the mantle of a true leader. You are becoming a leader who looks out for your peers and considers the impact of your code on customers (feature-set), the business (performance), and your community (developer maintainability). This multifaceted mindset towards your work is why your team now looks to you as the paragon of Quality in Software.

- Liam Yafuso, CTO at Artium