Remember Waterfall project management? When DevOps/Agile were the new kids on the block, software saw the light and all was going to be fixed (including the holy cows of specifications Bibles and Gantt planification)?

Alas, not all that shines is a panacea, DevOps principles have their own dark side and anti-patterns. Let’s be clear on our focus here, these are not the classic anti-patterns of DevOps washing or proto-Scrumfall  — which are all valid points but certainly not the subject here. So let’s assume these are sorted and take a clear step up above the basic pitfalls of DevOps transformation and see how much costs are still hidden here and there, even when we do it all by the book.

So what could go wrong? Well depending on where you stand in the organization, it could mean budget costs, time costs or frustration costs. Let’s review them.

As Software Engineers

We developers look up to DevOps as a great agility ecosystem, where making it work and value delivery beats code orthodoxy.

Sure, pushing features in production before deadlines is the adrenaline rush of the team — and at least we get to say something interesting in daily standup. Sure, DevOps principles allow for the ultimate problem solving mindset to rock the place. However piling up new tools only sign you up with documentation, maintenance and god-forbid, vendor lock-in.

So in the end, be wary not to over-engineer it (like, are you sure the company blog needs an in-house multi-region Kubernetes cluster with highly persistent volumes?) Startups more often die from drowning under technical debt than drying up from business opportunities.

The Painkiller: resist the urge to try new shiny tools every quarter. Proposed changes should go through time-boxed experiments to assess gains vs debt. Finally, it should be a shared decision to onboard new tooling.

As a Sysadmin

The appealing side of DevOps for Sysops/Platform Engineer and this crowd is, yes, you got it right, automation! Especially for SRE roles, being change-adverse by nature (because who wants to add instability in production systems?), their appeal for automating all the things cannot be more legit.

Removing the human factor from delivery and release, from scaling up & down to restarting unhealthy servers is priceless. It brings confidence, it builds bridges between dev and ops and ultimately frees up resources for higher value work. Many teams did that and found themselves surprised by the big bad bill at the end of the month. Well, it’s the other side of the same coin: spawning an additional production-like environment per branch is not cheap. Even storing hourly backups of production databases in multiple providers will eat  egress network bandwidth like crazy.

Because you can automate a process, does not mean you should abuse the robots and make it happen for every event at every hour of the day and night. That would surely blow up the budget, only to summon the accounting Department to your cubicle, shortly followed by a dance of tickets with AWS Billing Support.

The Painkiller: first things first, set Billing limits on the cloud provider accounts. Also, enabling reasonable quotas on resources, like maximum servers. Then, enable monitoring with alarms on these resources. Finally, take the time for an inventory of automatable tasks that will help sysadmins stabilize production the most.

As a Product Manager

Keeping track of things past and future is a tedious, yet unavoidable task. The Agile ecosystem can generate so many marketing and sales tricks, that even the most sincere DevOps practitioner will fall for the Swiss-knife of project management and collaboration. Atlassian Jira in this matter is somewhat of a legend.

Navigating Jira is like Asterix stuck in the bureaucracy maze. They move around a lot, with a weird feeling of not getting things done. Sure, the destination is at reach, and they move really really fast (even poor Idefix!). At some point though, we see doubt, fear and uncertainty infused by the overhead of paperwork to go over, just for tracking purposes. Jira implementations and bureaucracy are so close that someone, not me, carved the term: “Jireaucracy”, that sums it all.

That’s a lot of energy gone, maybe hours per month, multiply this per engineer and the amount of it can be translated into OPEX. Over time, the dollars turn into burnout, with the feeling of fighting against a tool designed for frustration.

The Painkiller: product managers need to simplify workflows and keep a product mindset, with value delivery first. Avoid project-type success metrics like delivery deadlines and burndown charts in favor of business outcomes and real-life metrics, like user retention and increased sales.

As a Manager

DevOps as an organisation project has been an open question from 2009 to this very day. Books have been written on the subject, among them the seminal work of Mathew Skelton on “DevOps Topologies” , describing how to name positions inside teams and their relationship according to products and projects.

Once again, the effort of adopting DevOps has a direct effect on budgets like human resources and training. Transforming people’s mindset and culture is the primary challenge. If a team resists change because of opaque information, inefficient handoffs, or uncontrolled perimeters, the organizational transformation is pretty much dead in the water.

The Painkiller: leverage senior roles to accelerate transformation. By promoting internal champions, they can show and tell about their successful experiments. Building momentum is then just a matter of how many people you can involve in DevOps proof-of-concepts.

TL;DR

So here we are, with hidden costs of DevOps in non-obvious steps. Next time you talk with a member of the IT department about going “full throttle on DevOps”, please remember these few words, as there are summed up below for your convenience:

Developer pitch

• Cross-functional teams should evaluate new tooling to avoid over-engineering. Shorter development cycles imply tradeoffs in quality, thus inflating the technical debt.

Sysadmin Pitch

• Automating delivery and deployment in the cloud requires precise measurement of billing impacts, ensuring additional costs are justified. Unused resources spawned on a “git push” basis can lead to self-inflicted “Denial of Wallet” attacks.

Product Manager pitch

• SDLC and their flows often end up struggling from “Jireaucracy Syndrom”, building new walls, impeding productivity, contrary to agile principles of DevOps.

Manager Pitch

• Hiring and training people with the right mindset is a huge investment, to be spent upfront. Once the right culture is infused in daily practices, only then DevOps can start.