Delivery process: from quantitative to qualitative analysis
Moving from quantitative to qualitative analysis
While quantitative analysis offers a solid base of facts to understand AGL's delivery process performance, numbers by themselves cannot lead to real changes in any organization. In addition, it is required:
A shared understanding of the obtained data
Clear ways to communicate the outputs
Simple and easy to apply ways to guide decisions supported by data, for different groups of people across the organization, from individual team members to leaders.
This is why moving from quantitative to qualitative analysis is important. By changing numbers into clear descriptions and helpful contexts, teams can agree on current process performance, find specific ways to improve processes and practices, set common goals, and define clear targets for future work cycles: continuous improvement.
This method turns hard-to-grasp metrics into easy-to-understand knowledge for the organization. It supports ongoing improvements based on data, which can be shared with and understood by both, technical and non-technical people.
The process explained below gives a step-by-step way to make this change, using AGL as example.
Process description
Moving from quantitative to qualitative analysis requires:
The definition of a threshold for each of the variables that define the delivery process performance metrics: throughput and stability
Taking such threshold as reference, the following step is defining trends, for values above and below of each of the previously defined thresholds.
Using a reference delivery process performance lifecycle, and based on the previously defined trends, reference scenarios will be created. We will use them to describe AGL’s delivery process performance qualitatively.
Once we have such reference scenarios described, we will determine which ones better describes AGL delivery process performance at any point in time, for instance, in 2025Q1. We call it current scenario.
Finally, we will define a target scenario for the following delivery process performance improvement cycle (continuous improvement process/kata), in our case, 2025Q2, which will be confirmed or not in a follow up iteration of this study.
In summary, when applying a data-supported continuous improvement process at scale, across an organization (or open source project), building a common understanding of where you are (current scenario) and where are you heading to (target scenario) is essential. Communicating both scenarios, not just quantitatively, but also qualitatively, has demonstrated to be very effective to improve the delivery process performance.
1.- Thresholds
The threshold values for each of the measured variables corresponding to the iteration 1 model, which AGL’s contributors have defined, are:
Please remember that the median correspond to the 50 percentile line and that IQR is defined in the Measurements and plots page.
Pick realistic values for each threshold instead of those who you would like to see in a far future, so hard to achieve. Making target scenarios achievable will be essential to execute continuous improvements processes successfully. And those target scenarios rely in these thresholds, as you will see.
2.- Trends
For each metric/variable, AGL defines a trend depending on getting values above or below the predefined threshold. You can see in this graph the thresholds and the designed trends for this particular case.
Define the thresholds and the trends for your own case. Use the above just as reference.
3.- Reference Scenarios
Once the trends are defined, it is time to define the possible scenarios. We recommend to define a life cycle for those scenarios. The scenarios might vary over time but life cycle should be fixed, in most cases.
This is the life cycle we use for AGL and for most automotive projects. The life cycle is formed by different states.
Our recommendation is that you define each state based on your OKRs and outcomes, not in KPIs or metrics.
With the above states in mind, let’s create the reference scenarios, based on the trends we have defined for each variable using the arbitrary threshold AGL has agreed on. These are the reference scenarios, based on the trends, corresponding to each one of the metrics.
Depending on the nature of the delivery process and its performance, these scenarios might not fully apply to your case. We have tailored them to AGL’s delivery process performance based on the current study. They might change over time.
As you can see, there is no scenario corresponding to the Excelling state at this point. I suggest you revisit the states definitions and the scenarios once you have gone through a few continuous improvement process (kata) iterations. In general, try not to define unachievable expectations nor using this exercise to finger pointing at teams. Remember, we are defining the delivery process performance, not teams or people performance.
You can find other reference scenarios applied to a different delivery process in a different environment and context in the book Measuring Continuous Delivery, which is referenced is in the Reads section.
4.- Current scenario
Checking at the delivery process performance dashboard page (first iteration), that is, Pd.1.1., P.d.1.2., P.d.1.3. and P.d.1.4., we agree that the trends during 2025Q1 could be described as in the picture below.
Checking at the reference scenarios, we can define the following:
The conclusion of this exercise is that, currently, based on the agreed thresholds by AGL, their overall delivery process is:
Throughput: Slow and infrequent
Stability: Fragile
5.- Target scenario
The target scenario, expressed qualitatively is: slow and reliable
What we are saying with this drawing is that, during the following improvements cycles, the goal is to take the delivery process to a scenario where AGL developer see:
The code flowing more frequently throughout the delivery process.
In terms of metrics and thresholds, the above is equivalent to say that the median of the Time Interval should stay below 5 days across 2025Q2 while keeping its IQR and the lead time IQR in the current values. Ideally, lead times should not escalate beyond the current values, although it is not the main focus.
The code flowing more reliably (with less interruptions) across the delivery process
In terms of metrics and thresholds, the above is equivalent to saying that Change Failure Rate should go below the 40% mark while keeping the current values of the median and IQR of the Failure Recovery Time.
Once you reach the expeditious and reliable scenario, it would be time to adapt you thresholds, redefine your scenarios and iterate.