Poor DevOps Kills Developer Productivity

Software development business has a special nuance to it which affects it dramatically. And most likely you came across it if you participated in the process of filling up positions of software developers. It is hard to do. And even if you succeed, it is not guaranteed that the person will not job hop after a year or two. The market is flipped, so there is actually more demand for developers than supply is ready to offer.

I want to talk about the ways of lowering the software dev job market pressure on the product. Agile and CI/CD practices came for a reason, and I believe that one part of this reason is reducing the impact and risk of poor code, as well as to raise developer’s efficiency. In my opinion, the CI/CD pipeline combined with tests comprise a reusable storage of collective experience about developing the product.

Let me elaborate on that.

A software product is much more than just a bunch of code. A product also includes all the wiring like tests, deployment process, infra configuration, documentation and even development process.

Again, all this wiring is a part of the product, and cannot be separated. It defines efficiency of the product itself and its ability to exist. Software is a living thing and it has no ‘stationary’ position. It is either developing or dying off - just like an organism. You can try to keep it in low maintenance mode, but it is just a matter of time when it will become financially absurd to keep it running, because the border of ‘low maintenance’ is ever expanding.

To me, efficient DevOps practices are all about this wiring around the product.

While HR processes are running on their own pace, there is so much more which can be done to lower the impact of an employee leaving or onboarding a non-experienced one. Lets try to list the biggest but commonly underestimated problems which entangle software like an invisible web, slowing everything down.

Many ways of developing things

Your organization might be lucky to have one platform for running your software environments, but most likely it is not. Usually it resembles this puzzle:

• If you develop component A1 for product A, then follow dev, test and deploy processes X
• If you develop component A2 for product A, then follow dev, test and deploy processes Y
• If you develop component B1 for product B, then follow dev, test and deploy processes Z, but keep in mind that you need to use process Z1 if you deploy to production
• … (you get the idea)

I am pretty sure many people can relate to that situation, because environment platforms develop organically and new features are introduced with new components. It is very hard to find priority to bring up all existing Infra components up to date with the latest components. Why? Simply because it is a very complex affair and requires many extra steps to perform with zero downtime (which is usually desired by business). Imagine developers deploying production code to a system with active workload - to me it looks like running software updates on a flying airplane. Now imagine electricians applying changes to physical hardware on a flying airplane. The complexity is suddenly increased exponentially, because you are changing underlying layers.

Even though it is a fun thing to do - juggle with HA systems, solve puzzles on handling non-HA stuff, introduce brief backend outages (which hopefully nobody notices) - reality is that this is usually not a business priority because it is hard to justify the benefits of the activity. And such activities require intense planning and testing.

Dev != Test != Prod

This one is obvious, but worth mentioning. Very often it is a real pain, while it is usually okay to have variations between development and production to have good development experience, test to production variations are bad as they introduce hidden knowledge which can only be learned by trial and error. Poor development setup is a tremendous slowdown on its own, because of many reasons. Just to name a few:

• Long time to setup environment and start working
• Not able to reproduce behavior from production
• Hard to switch between components for development

All these points are multiplied by the previously mentioned issue of having many ways of developing things. This is a huge productivity killer.

Poor (semi-manual) testing

I am talking about the whole test suite setup. How useful tests are? How good are the environments for running tests? Do they provide enough confidence that you can rely on them and follow accordingly? Everything which stands between test results and a decision to do a deployment to production is arcane knowledge, which prevents developers who have less experience with a given product from contributing to it fast and efficient. It makes switching between the products slow and painful. Manual checks which are performed during deployments all need to be coded to automated tests, because otherwise you are introducing human factor.

Of course, this comes with a trade off of maintaining these tests, but this is another topic for discussion - if the test is not worth maintaining, then maybe it is not worth having it at all? Or, saying it differently, if the check is so important, that a developer has to perform it during every deployment - usually it is cheaper to maintain a test for the check, than do the manual labor.

So what

Of course, a real list of things which slow down developers is much larger, but these are quite often the biggest gorillas which devs have to deal with daily.

It is important to have your development processes work as a well-oiled machine. It is a complex undertaking which is easier said than done, but doable with small, continuous improvements. What is the current status? How many platforms for running software you have? The fact is that the more clutter you have at the backstage of the shiny product, the slower you are. You also need more experienced developers if the CI/CD pipeline is half manual. A good wiring for software product can lower the barrier for newcomer developers and make existing developers more productive.

I would try to dig deeper on these topics in my upcoming posts.