By 
DevOps is a cultural and organizational transformation that facilitates collaboration between development and operations teams. This new way of working requires the organization to adopt agile practices such as continuous integration, continuous delivery, and constant feedback.
DevOps is a journey, not a destination. The goal of every organization should be to continually improve their delivery effectiveness and efficiency to achieve the fastest time-to-market with the highest quality possible.
However, measuring your performance can be challenging. It is often hard for teams to know what steps they need to take next toward better delivery performance – either from an organizational perspective or within each group itself.
This blog post will explore some key metrics that you can use to measure your DevOps success.
Pipeline Flow Metrics
The solution is only as good as the continuous delivery pipeline itself. If your pipeline is slow, error-prone, and difficult to use, then the solution value will be limited as well.
A fast and stable continuous delivery pipeline means that it takes much less time for teams to know about failures in the build and deployment process – and thus, they can fix them quickly or rollback if needed.
Different DevOps experts recommend other pipeline flow metrics. Google, for example, focuses on lead time and deployment frequency.
- Lead time – The time taken by a single change to progress through the entire pipeline. The faster your lead time, the better impact you will have on delivering business value. If it takes six months for one developer to go from an idea in their head all the way to seeing that idea live in production, then this means many changes are being held up or delayed unnecessarily.
- Deployment frequency – The number of times a particular change is deployed into production over a given period of time. This metric measures how often you are deploying changes that have been validated in your pipeline.
In other sources, you might find DevOps metrics such as flow velocity, flow efficiency, flow time, and similar. Some also recommend using change volume and work in progress volume:
- Change volume – The number of changes (commits) that are passing through the pipeline at a given time. This metric gives you an idea about how busy your pipeline is and might indicate potential bottlenecks.
- Work In Progress (WIP) volume – The number of items (changes, builds, deployments) in progress at a given time. This metric can help you understand whether your pipeline can handle the current load and identify potential bottlenecks. If the number of WIPs gets too high, you might need to expand your pipeline or stop taking in new changes until the existing ones are completed.
Regardless of which exact metrics you use, it is important to track the trend and improvements over time.
You should also monitor your pipeline for bottlenecks and potential areas of improvement. For example, if you find that most changes are blocked in the QA stage, you may need to reevaluate some things there.
Solution Quality Metrics
The DevOps culture emphasizes shifting technical practices to the left. What does this mean?
It means that testing should happen earlier in the process, not just at the end. This also implies that teams are more proactive about avoiding defects rather than just fixing them after being released into production.
The longer your software is being used by customers, the higher the risk of encountering bugs and other problems there will be over time. By shifting technical practices to the left, you reduce the time these problems impact your customers.
Teams should continuously invest in building quality into their solutions throughout all stages in their software development lifecycle.
An example of a key metric used to gauge the quality of the solution includes Google’s change failure rate.
- Change failure rate – This is the percentage of changes released to the customers that resulted in a degraded service and subsequently required some time of remediation. This metric tells you how frequently your solution is not meeting the quality bar. According to Google itself; this percentage ideally should be anywhere 0 – 15%.
Solution Value Metrics
A high-quality solution delivered quickly through a highly optimized pipeline means little if the solution does not provide value to the customer.
The challenge for most organizations is that it is difficult to determine how much business value they are delivering with their current solutions. Therefore, solution value metrics are important to track the improvements over time. They include both economic metrics and customer satisfaction and engagement metrics.
- Economic Metrics – This could include measurements such as net present value (NPV), return on investment (ROI), and break-even point.
- Customer Engagement Metrics – These are metrics that track how customers use the solution and whether they are getting value from it. Examples of customer engagement metrics include utilization rates, customer satisfaction ratings, and net promoter score (NPS).
For example, you might want to use customer ticket volume as a proxy for customer engagement.
- Customer ticket volume – This is the number of tickets (or support requests) generated by customers in a given time period. This metric gives you an idea about how engaged your customers are with the solution and whether they are finding value in it.
It is important to track the trend of these solution value metrics over time in order to be able to determine whether your DevOps initiative is resulting in increased business value for the company.
To follow the example of Google, time to restore is a good solution value metric.
- Time to restore – The time it takes the team to fix an issue in production or restore service once a user reports that they are having problems. By measuring this, you can ensure that your solution is working as intended and provides real customer value. As with other metrics, there should be a general improvement over time for this particular metric. Google recommends that businesses keep time to restore to under an hour.
Where to start?
We have certainly provided you with a lot of information on how to measure the success of your DevOps process. Where do you start?
One way is to identify one DevOps metric that aligns with a specific goal and track it over time. This will provide some insight into whether or not your efforts are showing progress towards reaching this goal.
For example, if your main goal is to reduce time-to-market, you should track the number of days it takes for a change request to go from commit to production.
Once that metric has been established and starts showing improvement over time, we recommend identifying another goal and starting on this process again with one new key performance indicator (KPI).
You will never have all the answers, and that is okay. As your company evolves, so too will your DevOps performance. The important thing is to be constantly learning, adapting, and improving.
What next?
Once you find the metrics that work for your business and have some data in your hands, what should you do next?
The next logical step is to start digging in and understanding what is underneath all those data points. What are the root causes of poor performance? Why are some solutions succeeding while others are not?
This is where root cause analysis (RCA) comes in. RCA is the process of determining the underlying reasons for a problem and then implementing corrective actions to address them.
It is important to know how to process the raw data coming in from your measurements.
This is where the power of data visualization comes into play. Visualization makes it easier to understand patterns and trends in your measurements that may not be immediately obvious when looking at raw numbers or charts by themselves.
It also provides a great way to share insights with others, which helps foster collaboration between teams and different business units. Data visualization should be a key part of this process if you want to get the full potential from your DevOps transformation.
Conclusion
You will never know whether or not your DevOps transformation is successful unless you can measure it.
This process can seem daunting at first, but having well-defined goals and metrics in place will make this much easier to accomplish. These metrics should encompass your pipeline flow, solution quality, and solution value. They should be in line with your company’s business goals and tracked over time..
Once you have some data, it is important to perform root cause analysis to determine the underlying reasons for any problems.
It would be best if you also took the time to implement good data visualization practices so that the insights gleaned from your measurements can be shared with others within your company more easily. This will help improve collaboration and communication between teams.
DevOps is all about working together to deliver more value to the customer faster. By measuring your success, you are taking a big step in the right direction.
Looking for a way to measure the effectiveness of your company’s DevOps transformation? Contact The i4 Group for more information on how we can help.
