Moving the needle and keeping it there

I hear the phrase “moving the needle” a lot when we’re speaking with customers and organisations about them wanting to embrace a DevOps methodology using automation. The origins of this phrase appear unclear – but the one I like refers to sound recording in old analogue recording studios. Here, the VU or “Volume Units” meter was used, a physical dial with a physical needle to detect the level of volume produced. Some audio sources were not loud enough to move the needle off its stop; in other words it was too faint to be heard. However – when the audio moved the needle, it was enough to be heard, enough for the listener to hear and to take note of, enough to make a difference.

The Effort Required

People talk about the effort required to move the needle, to create the ground-swell of energy and momentum required to show people a new way of working, and to bring them along on the journey. In our experience, building excitement and buy-in at this point is relatively easy. To get started, to demonstrate value and to show the art of the possible. At this early part of a DevOps implementation or transformation, everyone’s very excited about the potential and the possibilities. It’s new, it’s shiny and it’s delivering – by this point, the naysayers who said “Well, of course, that works there, but it won’t work here for reason A, B, C”, have been won over and in our experience can sometimes become the biggest champions. The people who sat on the fence to see how it would pan out are on board too; perhaps they were a little reluctant at first – but they can now see the value in it.

Moving the needle is exciting. It’s often visible throughout the whole company (or at least it should be!) and people buy into it. People are able to see the value DevOps brings. The comfort you get as a developer from being able to make a change – safe in the knowledge the impact is minimal rather than crossing your fingers and hoping all is well. Or through automating the deployment process knowing that nothing has been missed, or perhaps safe in the knowledge that if something does go wrong, you can rollback to the earlier release at the touch of a button. All of these benefits are well documented and understood.

The part people talk less about is how do we keep the needle there? What happens when the Agile consultants have finished their work and ridden off into the sunset… What happens when it’s not new anymore? How do you stop it from moving backwards? How do you stop people’s behaviours from reverting back to what they had before?

Tangible Changes

I recall working with a company where they made real, tangible changes to their processes, their people’s behaviours, mindset and their technology. They’d been given the breathing space to allow this process to happen, everyone top to bottom was fully bought into what was required – allowances were made, customer expectations and timescales adjusted to accommodate this change in working practices – with everyone bought into the vision and what it could deliver.

Reaping The Rewards

They worked hard and they reaped the rewards. Everything continued well for a while until the day when they had an important set of changes to meet a regulatory requirement. Everyone agreed that this had to be done and that teams were stretched. Both things were very important for different reasons – one had to be done but didn’t offer any advantage or benefit to the customer. The other offered lots of features many of their customers were asking for.

They could do one or the other “properly”, but both were really important… They’d proven this new way of working worked and everyone was bought into it. Could perhaps an exception be made just this once? Could we just develop both of them and have a light touch around unit testing and automated test coverage…? It would just be this once, and yep – everyone understood we’d accumulate technical debt – but it would just be this once. As soon as it was released, everyone could “shore it up” putting everything back in place and then move forward as before.

They did. They never recovered. It was the start of a steady decline. The builds were broken. The test coverage wasn’t there to give them the confidence all was well. They started to put more emphasis on exploratory testing – using it as a substitute for test automation. Confidence in the whole process eroded quickly… Confidence is a tricky thing. It takes a long time to build and no time at all to destroy.

This isn’t an isolated incident… We’ve come across a number of examples over the years where organisations have tried to embrace DevOps, starting with build and unit test automation, progressing through test automation and through to automated deployment. More often than not – the failure at any of these stages isn’t in implementing it – it’s in keeping it going.

How then do you prevent this? How do you hold the line, keep the focus and maintain the discipline? In our experience, the key is in having a zero-tolerance approach to deviation from the process, no corners cut, no technical debt accumulated. If you slip once, you’ll do it again – and each time you do, the path back becomes harder.