Why the Four Key Metrics Are Essential in Today's Software-Driven Automotive Industry

SECTION 1

Introduction: when the car becomes a software product

Today's cars are computer networks on wheels, often containing more than a hundred electronic control units (ECUs) and tens of millions of lines of code, sometimes more than some consumer operating systems. The rise of software-defined vehicles (SDVs) has made software a key differentiator for manufacturers, driving a significant reordering of industrial priorities. Many companies are bringing software development back in-house, recognizing its critical importance for achieving a competitive edge against natively software-driven players.

This shift from a hardware-driven development approach to a software-driven one presents considerable challenges. In the automotive sector, software and hardware integration typically happens late in the process, often close to production launch, a direct consequence of organizations historically structured around processes like ASPICE and platforms like AUTOSAR Classic, designed for a long V-cycle rather than continuous iteration.

SECTION 2

The late-integration trap

### The hidden costs of big-bang integration

Late integration leads to costly, recurring consequences: dedicated teams mobilized in emergency mode to fix defects found late, production schedule delays that ripple back to the assembly line, and trade-offs on quality or functional safety to meet rigid industrial deadlines. Development silos between hardware, software and systems teams, combined with limited automated test capacity, contribute to "big bang" integration, where problems surface too late to be addressed effectively without impacting the schedule.

These issues are compounded by the unavailability of target hardware during the early phases of software development: a production-grade ECU often only exists a few months before launch, making exhaustive testing on real target hardware practically impossible before that point.

### Shifting left, continuous integration and fast feedback loops

Late integration challenges aren't unique to the automotive sector. Similar problems held back software development in other domains until those industries adopted iterative development and continuous feedback. Three practices structure this transformation: shifting testing left, carrying it out throughout development rather than saving it for the end; continuous integration (CI), where automated systems check every code change to ensure early defect detection; and fast feedback loops, whose quick insights reduce surprises and enable timely fixes rather than late-stage rework.

These practices enable high-quality software delivery in the sectors that adopted them first, and their principles can be transposed to automotive development, provided they are adapted to the constraints of hardware and functional safety (ISO 26262, SOTIF).

@cite:shift-left-testing-benefices

SECTION 3

The four key DORA metrics: a framework for measuring performance

### The four metrics in brief

The four key metrics, introduced by Nicole Forsgren, Jez Humble and Gene Kim, and popularized since by Google Cloud's annual DORA (DevOps Research and Assessment) reports, prove very useful for measuring delivery performance. They are: lead time for changes (how quickly can a change be tested and validated?), deployment frequency (how often can teams deploy updates?), change failure rate (what percentage of deployments fail?), and mean time to recovery (how quickly can a team resolve an issue after a failure?).

@cite:metriques-de-fiabilite-logicielle

### Why they seem poorly suited to automotive

From an automotive standpoint, these metrics may initially seem poorly aligned, given the low frequency of launches, a production cycle spans several years, and a cultural mindset inherited from functional safety that things must be right the first time, under penalty of a costly product recall. However, when reframed to measure feedback loops and incremental progress rather than final production alone, they reveal areas for improvement even in the most hardware-constrained environments.

SECTION 4

Adapting the four metrics to hardware-constrained environments

### Redefining each metric stage by stage

Rather than tying these metrics strictly to final production, they can be adapted to reflect the best available environment at each stage, the one closest to real conditions. Lead time for changes then becomes the time needed to validate a change in the available quality assurance environments, such as Software in the Loop (SiL) or Hardware in the Loop (HiL). Deployment frequency measures how often teams check and validate their software in the best available environment approximating production. Change failure rate highlights the percentage of failed deployments, including failed validations and checks at earlier stages. Mean time to recovery tracks how quickly teams address issues identified during downstream validation, from SiL all the way to the complete vehicle.

### SiL and HiL: production by proxy

This approach emphasizes fast feedback loops regardless of the final delivery cadence, and encourages accountability across every validation stage, from local development to SiL, to HiL, all the way to the vehicle itself. Modern HiL rigs, often virtualized and remotely controllable through cloud platforms, now make it possible to run automated validation campaigns continuously rather than on a one-off booking of a shared physical rig, a change of scale that mechanically brings automotive DORA metrics closer to their pure-software counterparts.

SECTION 5

The hardware journey: from software tests to prototypes

### Five stages toward the target hardware

Achieving fast feedback loops in hardware-constrained environments requires a staged approach, since target hardware is often only available late in the project. It starts with software-only tests, run on developer machines or emulators like QEMU, validating compilation, static analysis and unit tests. Next comes the adoption of virtual ECUs, through tools like dSPACE VEOS or Synopsys Virtualizer, which enable integration testing in cloud environments before physical hardware is available. Teams then leverage evaluation boards, off-the-shelf hardware that simulates target environments and surfaces performance issues early. When evaluation boards aren't enough, they build mockware: custom mockups (CAN bus simulators, USB cameras, USB sensors) to test integration and gather relevant insights. Finally, the transition to prototypes progressively replaces mockups while carrying forward the benefits of earlier iterations.

### What should, and shouldn't,wait for the final hardware

This iterative progression ensures software quality and integration readiness improve well before the final hardware arrives. It lets teams validate core functionality early in the process: for example, if video processing is a key feature, it may not initially matter whether the camera is connected via USB instead of LVDS, or even whether it's the actual camera or simply an available camera with similar specifications. What matters at this stage is securing the software architecture and interface contracts, not full hardware fidelity.

SECTION 6

Walking skeletons, vertical slicing and automation

### A walking skeleton is not an MVP

To avoid the pitfalls of late integration, start with a walking skeleton, a minimal version of the system linking the key components with basic functionality. This foundational step should take days or weeks, not months. It lets teams ensure early integration between components, verify that pieces work individually and together, and build incrementally by adding features afterward. Paired with vertical slicing, breaking problems down into smaller, self-contained, end-to-end shippable pieces, this approach simplifies integration and debugging, reducing defect rates and overall costs. It's worth noting that a walking skeleton is not a minimum viable product: if the team still finds itself thinking in terms of months of development, the idea needs to be scaled down further to its essentials.

### The central role of automation

While early feedback can't come from serial-production hardware, it remains invaluable for guiding development and reducing surprises. Automated tests, built alongside production code, ensure fast validation of code functionality and architectural alignment, integration with surrounding systems, and non-functional requirements such as performance and latency. Early validation builds confidence, surfaces problems before they escalate, and shifts quality improvements earlier in the process. Automation, CI/CD pipelines, HiL campaign orchestration, automatic generation of ASPICE compliance reports, is the cornerstone of this approach, enabling faster iterations and continuous architecture adjustments to overall needs.

@cite:pipelines-cicd-auto-reparants-self-healing

SECTION 7

Measuring at the system level: conclusion

Focusing solely on team-level metrics risks producing local optimizations that obscure the bigger picture. In automotive, where a single deployment often affects several interdependent ECUs, it's essential to measure performance at the system level rather than at the level of each isolated component. By emphasizing overall system readiness and integration quality, teams can align their efforts toward cohesive solutions rather than isolated wins; measuring lead time and recovery time at the system level reveals bottlenecks and fosters collaboration across historically siloed teams.

The four key metrics thus provide a valuable framework for improving software delivery performance in the automotive industry. By adapting them to hardware-constrained environments, emphasizing early feedback, and leveraging iterative practices like walking skeletons, teams can deliver high-quality systems more efficiently. The path toward better delivery performance starts with small, incremental changes: testing vertical slicing on a single subsystem, or instrumenting a first SiL loop, is often enough to build a foundation for long-term success. As the automotive industry fully embraces software-driven development, these principles help organizations thrive in a fast-evolving landscape.

Disclaimer: The statements and opinions expressed in this article are those of the author and do not necessarily reflect Adservio's positions.

FAQ

Frequently asked questions