Introduction

This analysis is focused on describing what the data is telling us about AGL. The result of this descriptive analytics stage should be:

• Conclusions that help us to characterize AGL through data

• Questions that will help us to narrow the scope of the diagnostic analytics stage, which should take place as a second iteration of this study.

This page reflect the most relevant points that have called our attention out of the data and dashboard pages. We have structured the descriptive analysis in three different areas of study:

• Community: we have not put much emphasis on this area at this point

• Activity: we measured activity at AGL by looking at several data types:

  • Repositories and commits: code, documentation…

  • Builds: commits validation, integration, releases, yocto artifacts…

• Processes: we have analysed two processes

  • Code review process: we studied the nature and key characteristics of this process

  • Delivery process: we have studied the performance of this process

The observation duration is spans from 2023-06-01 to 2025-06-01

1.- Community

Checking the activity per time zones (A.e.5.) it becomes evident that AGL has contributors from all over the planet, being Europe the main geographic area.

AGL’s leading organizations when it comes to code contributions(A.o.2), responsible of 87% of the commits

1. Konsulko

2. Linux Foundation (AGL’s staff)

3. Collabora

4. AISIN

5. Toyota

The top three organizations of the above list are also responsible for 72% of the changeset submissions and, of around the 77% of the code reviews (graph not included).

Other relevant organizations are (in alphabetic order): Fujitsu, ICS, Igalia, Panasonic, Virtual Open Systems, …

The number of contributions by people not affiliated to organizations is not significant. AGL’s is an open source project developed by professionals, under working hours, as activity related graphs show, especially during global holiday times and weekends.

The overall number of unique contributors (authors) has experience a decline in 2025Q1 compared to 2024.

• This decline is more noticeable in non-core repositories.

• When it comes to core projects, the decline is more noticeable in meta-agl-demo and meta agl-devel repositories, compared to meta-agl (A.o.3)

2.- Activity

2.1.- Repositories and commits

Key repositories. These are the repositories across the AGL project that attracts more activity (A.c.4) and community members (A.o.1.):

• The first column correspond to those repositories that concentrate highest levels of activity (commits), the second column to those repositories that concentrate the highest number of contributors (authors), while the third column correspond to those repositories where the greater number of organizations are involved, through their affiliated contributors.

• These shows that the core activity takes places in integration activities.

The overall number of merged commits over time tells us:

• During February 2024, AGL experienced a high peak in commits merged due to modifications in the tickets/MR GitLab templates across all the repositories (A.c.1)

• The following graph (A.c.2) reflects the activity without considering .md files. You can see the impact of modifying the templates have on February 2024. Overall, around 10% of the commits modify .md files.

  • Remember that there are different types of files with that .md format/extension.

• The lowest point was reached during February 2025

• The number of merged commits is declining in 2025 compared with 2024.

• AGL’s participation at Embedded World 2024 could have been a strong motivation factor for the project, which led to the highest overall activity (merged commits) peak (A.e.1.).

The number of merged commits in specific repositories over time (A.c.5) tell us:

• meta-agl is not getting in 2025 the level of merged commits reached at the beginning of the observation Duration or in the April to July 2024 time frame.

• The variability in merged commits to meta-agl-demo and meta-agl-devel is high.

• The peak in the overall merged commits during February 2024 respond to a increase in activity in meta-agl, but specially, in meta-agl-demo

• The documentation repository gets higher levels of “attention“ around releases.

  • It did not get a similar level of attention at the end of the observability duration time, (Quirky Quillback release in May 2025) compared to other periods (previous releases).

• After noticeable activity in the compositor repository during 2024Q4, the component seems to reach a maintenance mode in 2025.

  • The analysis can highlight this transition (from development to maintenance) easily, by analysing the behavior of different activity metrics on any repository.

  • Repositories like ci-management and agl-service-hvac have increased the number of merged commits in 2025 compared to previous years, within the observation duration period

We analise now possible relations between the evolution over time of the number of merged commits, the organization (AGL) events and product milestones(releases):

• The time between CES 2024 and EmbeddedWorld 2024 shows a heavy growth in the commit merged(A.e.1.) to all three AGL platform flavors (meta-agl, meta-agl-demo and meta-agl-devel).

  • It is worth evaluating if there is a relation.

  • Please remember the AGL-repo correspond to the Yocto manifest

  • The same evaluation should be made for the commit merged in meta-agl(A.e.2.).

  • The peak in both graphs(merged commits considering all the integration repositories and considering meta-agl only) and their behavior (not the values) in 2024Q4 and 2025Q1 are similar.

2.2- Builds

The variability in the behavior of the successful builds over time graph is surprisingly high (A.b.2.). The peak took place in November-December 2023 while the lowest point took place in January 2025

The successful build rate (A.b.3.)during the observation duration period is extremely high given the nature and complexity of the system AGL builds, especially considering that the system is built for several architectures. This successful build rate is slightly lower during 2025.

In terms of build power assigned (A.b.5), meta-agl/UCB is the main AGL focus, looking at the overall number of builds during the observation duration (again, between 2023-06-01 and 2025-06-01). Most of these builds correspond to meta-agl, including support for the official platform/hardware, to create UCB (Unified Code Base):

• QEMU for both architectures, x86 and ARM

• RPi

• RCar, from Renesas

A second bulk of builds corresponds to building meta-agl-demo for QEMU, in this case, for the x64_64 architecture. Building the documentation is also relevant in terms of the number of builds executed.

When analysing the build duration or build time (A.b.6. and A.b.7.) of successful builds, it is easy to see that build times have been increasing consistently over time up until 2025-02-28, when a new NFS service was implemented by AGL. The real impact of this measure will be evident after summer 2025.

The default builds for the four core repositories are among the top ten with longer build times, with over 2:30 hours (median) each:

• ci-platform-meta-agl-devel-verify: 3:41 hours

• ci-platform-meta-agl-verify: 2:40 hours

• ci-platform-AGL-repo-verify: 2:39 hours

• ci-platform-meta-agl-demo-verify: 2:37 hours

The total duration of all the builds provides an idea of the overall process time, which can be translated into costs for this kind of fully automated process.

In AGL case, unsurprisingly, we find the snapshots that integrates the code coming from the core repositories in the highest positions of the top 10 list, although not in the same position:

• ci-platform-meta-agl-verify: 59 03:04 days

• ci-platform-meta-agl-devel-verify: 54 01:09 days

• ci-platform-meta-agl-demo-verify: 50 22:44 days

• ci-platform-AGL-repo-verify: 48 10:21 days

In addition to the above, among the top ten we find:

• The sstate mirror builds (build-machines-sstate-mirror and sandbox-build-machines-sstate-mirror ) which are mechanisms associated to Yocto

• The verify-CIB-flutter-qemyx86-64 builds for meta-agl and meta-agl-demo/devel repositories as well as AGL-repo without flutter. Please remember that QEMU for x86_64 architecture is the default one at AGL during the observation duration period.

AGL has built (A.b.8):

• meta-agl (not counting agl-demo and agl-devel, which also include meta-agl) over 6.3 thousand times during the observation duration, including the different supported platform BSPs, which means close to six and a half months of build time.

• meta-agl-demo over 5.5 thousand times during the observation period, including the different supported platform BSPs, which means a little over five and a half months of build time.

• meta-agl-devel over 5.1 thousand times during the observation period, including the different supported platform BSPs, which means close to five and a half months of build time.

3.- Processes

Evaluation of the following processes:

1. Code review: we work on identifying the nature and key characteristics of the code review process

2. Delivery: we focus on studying the delivery process performance

   1. Please bare in mind that the release and the maintenance processes are out of the scope of this study.

   2. We faced limitation on the observability of the process which lead to some partial description in some aspects of the study this process performance.

3.1.- Code Review Process

Time to merge (TTM)

Analyzing the time to merge the patchsets (P.c.1.), we can see that the review process lead time is lower in meta-agl than in the other two main repositories of the AGL PLatform: meta-agl-demo and meta-agl-devel.

• meta-agl-demo has outliers that cause the 95th percentile to be high.

Analysing the behavior of this TTM metric over time, we can see in P.c.2. an ascending average trend, including three picks where, in a reduced amount of time, the avg. lead time multiplies by ten (April 2024), quadruples (July 2024) and triples (November 2024), respectively, including an ascending behavior at the beginning of 2025Q2, that might mimic these increases.

• These increases might be perceived as the result of applying a specific policy, a focused action to close pull/merge requests, or the action of a bot. This perception might have a solid basis in commercial environments.

• At AGL, like in many other open source projects, high levels of variability might respond to an organic behavior, triggered by:

  • A post- or pre-release period

  • The participation of the project at a trade show or event

  • A vacation period

• As the activity varies significantly from one week to the following one, the variability, when the code review process is measured in monthly periods, can also be high, like in this case.

• From February to April, where both, the TTM and the REI show a prominent peak in the behavior of the corresponding graphs, the following events and milestones took place:

  • q. quillback release: 2024-02-22

  • CES 2024, Las Vegas, US: 2024-01-09/12

  • AGL AMM 2024 Winter/Spring, Tokyo, JP: 2024-02-27/28

  • Embedded World(EW) 2024, Nuremberg, GE: 2024-04-09/11

• CES and EW are the two major events of the year for AGL

  • The activity in February 2024 reached high levels: A.c.2.

When focusing on meta-agl only(P.c.3.), we see:

• A descending trend over time of the TTM metric

• Lower values in general than when analysing the aggregation of every repo’s changesets

• Greater variability, in general.

• It calls the attention the behavior in 2025Q2, which could be similar to what was experienced during the same period in 2024. This behavior present correlation with the general trend of the integration repositories.

  • As previously mentioned in this analysis, the behavior of the TTM metric beyond 2025Q1 will require an extension of the observability duration, which will be carried out in a coming iteration of this analysis.

First time to respond (TTFR)

The analysis of the TTFR values (P.c.4) bring us to conclude that meta-agl gets the highest levels of attention from the AGL community, out of the main repositories (meta-agl, meta-agl-demo and meta-agl-devel).

Combining the analysis of this metric with the TTM values, you can explain the average values being, in general, significantly higher than the median values for every repository.

The median value of the FTR of meta-agl is higher than the median value for the TTM for the same repository. This could seem contradictory. It is not. The TTM is measure once the changeset is closed (abandoned or merge) so the existing changesets that are open do not count for this metric.

• This might indicate that there are outliers still open that should be managed.

It called our attention that the number of chengeset submitters to meta-agl-demo is higher than meta-agl.

Developers behind the flutter based AGL’s homescreen seem to do particularly good in this metric.

Review Iteration Count (RIC)

There is not much difference on this metrics across the AGL target repos(P.c.1.). The values are, in general, low. Some factors could influence these values:

• The level of knowledge of the contributors to these repositories makes the submitters highly trusted

• The general trust on the pipelines and the testing included in high. The confidence that the pre-commit testing on hardware will catch errors, for instance, is high?

The evolution of the patchsets per changesets over time(P.c.6.) reflect the breaks around Christmas time in 2025. On 2024 we see idle time during the second week of January, which is the week after CES 2024, where AGL did a relevant showcase.

We can see an unusual peak during 2024w46, that does not seem to have any direct relation with events or product milestones.

The decrease in 2025Q1 of the number of changesets per patchset could be due to…

• A reduction of activity in some of the repositories where the RIC was higher?

• An improvement on this metric across several repositories?

Review Efficiency Index (REI)

The Moving Average of the REI , tracking all repositories (red line at P.c.7.), shows that the backlog increased:

• Between Jul’23 and Sep'23

• Around the beginning of Feb’24

• Around mid Jul’24 up to Nov’24

It also shows that there are two significant periods where the backlog decreases:

• From mar’24 to Jun’24

• From mid Jan’25 until the end of the observation duration time.

The average REI(grey line at P.c.7.) show an anomalous increase between Feb’24 and May’24 that correlates with the peak highlighted in the Time To Merge graph(P.c.2.), in April 2024.

• This reduction of the backlog might not seem organic but all the evidence tells us it is.

• This behavior is affected, as in the case of TTM, by the influence that organization events and product milestones has on AGL development and delivery.

  • Please remeber that the AGL release milestones has a dependency with the Yocto project release cycle.

It is also noticeable a high variability increase, which started around mid Aug’24 up to March’25.

• Data from 2025Q2 indicates that this variability in the behavior of Avg REI is not increasing anymore.

  • We will need to confirm it once we increase the observation duration time.

• This period of increasing variability shows no correlation with the TTM behavior for the same repository.

Checking the moving avg. REI (red line of at P.c.8.) we can identify:

• Two significant periods where the AGL community backlog increased:

  • From Jun’23 to Oct'23

  • From Oct’24 to Jan’25

• Two significant periods where the backlog decreased:

  • From March’24 to May’24

  • From mid Jan’25 to mid March’25

• Those periods where the meta-agl backlog decreased(moving Avg. REI), the general backlog of AGL project also decreased.

The avg. REI at meta-AGL(grey line at P.c.8.) experimented and increasing variability starting in mid Jun’24 that reached a peak in mid Feb’25.

• Since then, this variability in the avg. REI from meta-agl seems to be decreasing., which will need to be confirmed once the observation duration period expands beyond 2025-06-01

Others

Checking the number of changesets and patchsets per repository(P.c.9.), focusing on the repositories with more activity throughout the observation duration period, it is no surprise by now that meta-agl is the repository concentrating a bigger number of of patchsets, changesets and approvals.

When it comes to voting (approvals), the repository that concentrates more attention per changeset is the documentation. This repository get higher levels of attention right before the releases.

When analyzing the evolution of the number of changeset submitters (P.c.10.), it seems that the last quarter of each year is where the number increases, while Q2 experiments a descent. It will be interesting to see if this pattern is confirmed during 2025.

Finally, we analise the evolution of the number of patchsets overtime

• We can see that, across all repositories(P.c.11.), there is a drop in the number of changesets during 2025Q1 and a peak during 2023Q4

• When we focus an meta-agl(P.c.12.), we see a different behavior, concentrating high numbers of changesets between May´24 and Jul´24 and between Dec´24 and Jan´25

3.2.- Delivery Process Performance

3.2.1.- Delivery process model iteration 1

Looking at the iteration one of the model, where the overall delivery process is considered as a black box, we can see:

Throughput

Looking at trends (6 months sample period) we can see that Throughput has decreased, specially during 2024 due to:

• An increase of the median of the lead time(P.t.1) while the dispersion of the values is slightly decreasing, showing a stable behavior during 2025Q1, while the dispersion(IQR) is slowly declining.

  • The tendency shown by the lead time when we zoom out, plotting the lead time using one year as sample period(P.t.4.), confirms the first part of the above statement. We will need to increase the observation duration time in one year to confirm that stabilization during 2025

• An increase of the median of the time interval (P.t.2.) while the dispersion remains approximately constant. During 2024Q4, a decrease in the time interval started that is becoming evident in 2025Q1.

  • If the current tendency is confirmed, the median of the time interval would reach, during 2025Q2, the values shown at the beginning of the current observation duration period, in June 2023.

Zooming in (sampling period of 3 months) we can identify:

• 3 clear throughput cycles during the observation time duration, reflected in both, the lead time and the time interval(P.d.1.1. and P.d.1.2.).

  • The median of the lead time (P.d.1.1.), measured on 3 months periods, has varied from close to a day to around 7 days, during the observation time duration.

    • The peak median values of such cycles increases over time.

  • The median of the time interval (P.d.1.2.), again, measured on 3 months periods, has vary from around one day up to eight and a half days, during the observation time duration.

• AGL’s delivery process performance throughput (time lead time and time interval) show a positive behavior during 2025Q1, confirming the high level trends

• This cyclical behavior has underlying factors that might affect or explain it:

  • Given AGL’s integration nature and a well established release cycle (every six months).

    • AGL platform is based on Yocto, which also have a well established release cadence

  • For AGL, specific trade shows are relevant enough to influence the overall activity. The same applies to AGL’s face-to-face meetings

Adding the events organization’s events and product milestones to evaluate correlations between the behaviour of the lead time and those events we conclude:

• Sample period of 3 months(P.d.1.1.e.): there is no relation, correlation or conclusions

• Sample period of 1 year(P.t.4.e.) hints a tendency change around CES 2024, worth exploring

Stability

A high level trend of the median of the Failure Recovery Time (P.t.3.) show an increasing tendency, while the dispersion (IQR) decreases.

Zooming in, FRT (P.d.1.3.) behavior is cyclic, with three cycles during the observation duration, just like the median of both, the lead time and the time interval.

• Median values of the Failure Recovery Time go from half a day up to three days.

• FTR tendency improved during 2025Q1, reaching lower values than at the beginning of the observation duration time (June 2023).

Change Failure Rate stays below 50% during 2025Q1.

• The CFR behavior shows high variability, with a particularly high value around March 2024

• Overall, CFR values are high.

3.2.2.- Delivery process model iteration 2

Looking at AGL’s delivery process model iteration two, where the entire process is divided into two stages, commit stage and integration stage, we can see:

3.2.2.1.- Commit stage

The analysis of the delivery process performance at this stage does not include the metric stability. Due to limitations on the observability of the system, it is not possible to associate failures to process triggers.

Throughput

Checking the commit lead time graph(P.d.2.1.), we can see:

• Values increased consistently during the second half of 2023, specially during the last quarter. The median of the lead time reached a peak of around 8d12:00 at the beginning of 2024

• After a decrease, there is a period of around 6 months where AGL managed to keep commit lead times(P.d.2.1.) under control, with low values.

  • Values quickly decrease to their all time minimum (across the sampling duration time) by April 2024: around 12:00 hours

• The median and the dispersion (IQR) of the commit lead time increased during 2024Q4 and 2025.

• The patterns exhibited by any of the two throughput variables at the commit stage do not correlate with the behavior shown by throughput at iteration 1.

Checking the commit time interval graph(P.d.2.2.), we can see:

• A peak for a median value of around 18 days and a minimum value of around 1d 18:00

• The behavior during 2023 is analogous to the commit lead time.

• A strong increase is perceived during 2025 which needs to be confirmed when the observation duration increases.

3.2.2.2.- Integration stage

Looking at the dashboards and plots corresponding to the delivery process performance at the integration stage, we can see the following:

Throughput

The behavior of the median of the integration lead time (P.d.2.3.) show high levels of stability, with values between around 8 and 12 hours throughout 2025Q1. The amplitude of the integration lead time values(IQR) is the smallest at this study, with maximum values of 5 days. The 75th percentile shows cycles of increasing peaks, which translate into increasing levels of variability.

The median of the integration time interval (P.d.2.4.), shows a behavior that correlates with the trend shown by both medians, from lead time (P.d.1.1) and time interval (P.d.1.2.) at the iteration 1 of the delivery process model, where we see 3 cycles, with increasing peak values. There is an abnormal increase of the 75th percentile around March 2024.

Stability

The behavior of the median of the integration failure recovery time variable (P.d.2.5.) correlates with the FRT behavior during iteration 1 (P.d.1.3.), showing three cycles with increasing peak values. That same correlation applies to the behavior of the median of the integration change failure rate P.d.2.6. variable and the CFR median (P.d.1.4.) at iteration 1, which is not unsurprising given that the CFR of the commit stage has not been considered in this study. During 2025Q1, the FRT is decreasing in a consistent manner. Finally, it calls the attention the significant increase of the 75th percentile of the integration FRT around December 2023.

The variability shown by the Integration CFR (P.d.2.6.), is a behavior to pay attention to. It goes from a minimum of 32%, around May 25th 2024, to a maximum of 77% by the end of February 2024

Next steps

This description of the most relevant aspects of the graphs has been complemented with conversations with AGL core contributors. The first goal has been to confirm that the data and the graphs are correct. The second goal has been to confront the reality described by the data with the what AGL contributors experience, identifying discrepancies, gaps, unknowns…

As a result of these conversations, which is an ongoing task:

• A series of questions, that should be considered as initial step for a second iteration of the study, are written in the corresponding section.

• The authors will provide some recommendations that go, from points for improvements, to areas where to pay more attention or focus more capacity.

• Provide a report.

These actions are not sequential and will be carried out in parallel with the communication of the results, internally at AGL, and externally.