How to Measure DevOps Success

DevOps is an approach to executing the software development lifecycle (SDLC) that makes collaboration a priority. It encourages Development and Operations teams to work closely together (sometimes combining them into one team) so they can provide continuous feedback and improvement at all stages of the SDLC, from software development to software deployment and service operation.

Adopting DevOps practices will ultimately help your business bring projects to life. However, many teams struggle to properly measure and articulate the success of their DevOps initiatives. This is especially true when an organisation is attempting a DevOps approach to the SDLC for the first time. Not being able to show the positive results of DevOps transformation can damage the relationship between IT and other company teams as well as cloud the impact IT has on business results.

There are various ways you can measure DevOps success which Ten10’s Cloud & DevOps consultants use all the time. Below are the most effective metrics you can measure:

DevOps success KPIs

Mean Time to Recovery

One of the most common metrics that can measure DevOps success is Mean Time to Recovery (MTTR) – the average time it takes for a system to become operational after failing. Lowering your MTTR has many obvious benefits including faster issue resolution and improved customer experience but quantifying your system downtime means you can estimate the cost of each system’s failure for your business (for example, when a system fail means breaching a customer SLA). Knowing the MTTR of all your systems can also help your risk analysis and determine which systems need to be made as resilient as possible.

This level of performance may conversely be measured or expressed as Mean Time to Failure (MTTF) or Mean Time Between Failure (MTBF), capturing the average time between system failures.

Cycle time vs lead time

Cycle time is the time it takes from picking up a work item to completing it. Lead time is the total time it takes from the work item being created to it being completed. Measuring cycle time against lead time can show you whether items are being finished in your current sprint setup or if they need to be broken down further to be completed on time. This can have the knock-on effect of making your development team’s workload more manageable, reducing their stress and increasing their job satisfaction.

Velocity

Another metric that helps manage team workload is velocity, which uses a Fibonacci scale to assign point values to user stories. Knowing your development team’s velocity – the total amount of points that can be completed by the team in a given sprint – ensures that team members are not allocated more work than they can manage.

Velocity also helps senior stakeholders make important decisions about what work should be prioritised by creating a numerical budget of points to ‘spend’ for each sprint. This means that although a team may only complete two user stories in a given sprint, knowing they are working on the two highest-value stories of a project backlog illustrates how productive the sprint was.

Cyclomatic Complexity

When assessing your DevOps pipeline, you may use cyclomatic complexity to determine how confident your codebase is. Expressed as a numerical value, lowering your cyclomatic complexity score means making units of code more independent and reducing the risk they post to your business operations.

Code can become more complex as systems grow and new features are launched. Business stakeholders outside of the development team may push to grow their systems without paying mind to how complex they are making it (simply believing they are ‘bolting on’ new features to stables foundation of code). Successfully reducing this complexity can make testing easier and the overall system more stable.

Non-statistical measures of DevOps success

Organisational size and project requirements may mean you have a range of additional KPIs to measure the success of your DevOps transformation but you should not lose sight of the non-statistical benefits of improving your pipeline.

Measuring areas such as cycle time and velocity should be done to improve work allocation and resource in your development team. The benefits of creating an environment founded on trust and transparency cannot be understated. Team members who have opportunities to share what has/hasn’t worked well and contribute to their workflow structure will be more engaged and satisfied with their jobs, which in turn should lower team attrition levels.

Improving your DevOps pipeline should also increase understanding and communication between the development team and the wider organisation. Senior leadership teams should be able to guide development teams when choosing project priorities (based on velocity) and estimating business costs (based on MTTR). This reduces the perception that development is a siloed department and minimises misalignment brought by unclear briefing information or a change of priorities.

Misconceptions of DevOps success

Before you start any initiative to improve your DevOps pipeline, there are two common misconceptions that you should be aware of:

Misconception 1: You can achieve DevOps success in isolation

Many people believe that DevOps improvements solely affect the delivery team and that, effectively, DevOps is driven ‘from the bottom up’ (making changes in your development team that benefit other departments and teams above them). Improving your DevOps is not a matter of scrutinising and changing your development team – it is about adopting a wider, holistic approach that means your development cycle works for all teams involved.

Clear communication between all teams is paramount. Otherwise, you risk making your development team faster and more efficient but, ultimately, out of sync with other departments’ processes. For example, if a company’s development and security teams do not improve the DevOps pipeline together, the development team may be hindered by the security team’s longer task backlog. This masks the improvements you’ve made to the DevOps pipeline and only creates more problems. Ensure that your approach is ‘top down’ and that other teams participate in the pipeline’s transformation.

Misconception 2: Visibility = Success

DevOps should also not be seen as a one-off exercise to improve the visibility of your processes and results. Yes, establishing KPIs can help you determine where your problems lie and what ‘success’ looks like but beware of paralysis by analysis.

DevOps success is a process of continuous refinement and that means taking action year-round rather than simply when you encounter a problem. This is another reason why we emphasise cross-team communication as part of your DevOps transformation – balancing each team’s priorities as your business grows means finding incremental gains in your DevOps efficiency, rather than looking for the ‘silver bullet’ that will solve everything.

Why measuring success matters

Accurately measuring your DevOps success ultimately means being able to make informed decisions with better knowledge. Your development team can be transparent about where their resources are used and when they can deliver new features or functionality.

At an organisational level, you can better manage the costs associated with your development team by comparing KPI performance before and after pipeline changes. Automating this reporting and creating a dashboard you can use to quickly see performance levels means you can be proactive with the decisions you make. Just remember that true DevOps success is a continuous process of refinement and not every benefit can be measured and reported.