In the tech lead roundtables I join on software engineering, one question keeps coming up: where does QAware stand in its AI transition, and how do you even measure that? This article answers the second half of the question.
At QAware, we don't believe the shift to agentic software engineering can be measured as a progress bar that fills up to 100 percent. First, you are never done. Second, maturity has two sides: what the technology can do, and what the organization makes of it. A team can have coding agents, a well-maintained harness (the tooling and process environment around the agents) and access to frontier models, and still most of the team keeps working in a chat window. Another team has a strong review culture, clear specs and a firm will to put agents to productive use, but lacks proper access to the right LLMs or tools.
A conventional stage-based maturity model asks how a team works and gives both teams the same stage. Yet their problems point in opposite directions. Prescribe the same remedy for both, and you are wrong at least once.
For our own transition, we built a model that makes this difference visible. We use it regularly across our teams, and the results feed into our AI4SE status overview. Here we share how it works, what we have learned from it and where its limits are.
Stage models for AI in software engineering have been circulating for a while. Dan Shapiro describes six levels, borrowed from the levels of driving automation. They run from Level 0, your parents' Volvo, to Level 5, the dark factory: specs go in, software comes out. Steve Yegge describes eight levels, from programming without AI to orchestrating dozens of agents.
Both models are useful because they create a shared vocabulary. But both describe how an individual works with AI. The organization around that person is not captured as an axis of its own. They answer how far along you are. They don't tell you whether what is missing sits with your teams or in your toolbox.
Our matrix therefore separates technological maturity from organizational maturity. On the technology axis we distinguish three levels. In Assisted, the human writes and AI helps. In Agentic, the agent executes and the human reviews. In Orchestrated, humans steer several agents and their loops. The organizational axis shows how firmly working with agents is anchored in the company. It starts with Enabled, where the company provides tools and individuals use them on their own initiative. In Incentivized, the company actively promotes their use; in Adopted, it is the team standard across the entire engineering process. In Transformed, the way the team creates value is built around agents too: how it plans, what it commits to and how it measures success. For a services firm that means cost estimation and proposals; for a product team, roadmap and capacity planning.
Agentic software engineering maturity matrix
On the diagonal, shown in green, technology and organization match. That is balance, not a seal of quality. Individual assist sits there as well: individuals use AI as an assistant. That is a legitimate starting point. The matrix gets interesting away from the diagonal. That is where two kinds of gaps sit, and each name tells you what is missing.
Above the diagonal, in orange, lies the adoption gap: the technology is ahead of the organization. The tools are there, but the team and the company have not yet taken them up broadly. The cells are called Lone pioneers, Shadow automation and Tech frontrunner. Typically, the know-how rests with two or three people, and agent loops run without the backing of the team or management. If one of those people becomes unavailable, the team loses that capability. The remedy here is organizational: building skills more broadly, shared standards, a shared harness, clear roles and incentives.
Below the diagonal, in blue, lies the tooling gap: the organization is ahead of the technology. The will and the support are there, often even commitments to management or customers; what is missing are the right tools and the experience to use them. In a Prompt campaign, the company promotes AI but only provides chat. In a Chat team, everyone prompts and nobody has a harness. In Harness-bound, the way the team creates value is already built around agents, while the technology lags a level behind. The Overcommitted cell poses a risk for company leadership. The company has planned for agentic speed, in proposals or in the roadmap, but the team works with chat assistance. That is no longer a maturity problem, it is a risk to margins and commitments. The remedy here lies in technology and skills: Harness Engineering, Context Engineering, platform.
The diagnosis sounds trivial, but it has consequences. Launch a change campaign in the tooling gap, and you undermine people’s motivation who have wanted this all along. Keep treating the adoption gap as a technology problem, and you feed the shadow automation.
A colleague put it to me bluntly: a maturity model only becomes useful when a team rates itself lower than it feels. I would put it more gently, but he is right. Maturity models have a bad reputation, and it is not entirely undeserved. Too often they read like a certificate of what has already been achieved.
Research is skeptical, too. As early as 1984, John King and Kenneth Kraemer criticized the most-cited IT stage model of its day for setting an end state without explaining how to get there. The team behind "Accelerate" and the DORA research advises against maturity levels and favors capability models, because stages suggest an end point at which an organization is done. And literature reviews in computer science research have long complained that hardly any maturity model has been empirically validated.
Ours hasn't been either. We make no claim to measurement precision. The matrix is meant to create transparency, start conversations and help plan the next steps. It is explicitly not a tool for judging teams. A low rating marks the starting point of the next stage; a high one, an interim result.
That is why teams rate themselves. It works for us because they hold themselves to high standards and because a low rating has no negative consequences for the team. Both are preconditions. As soon as a rating decides budgets or performance reviews, gaming the numbers pays off, and the model ends up measuring tactics. Where that trust is missing,a central team should conduct the assessment, not the team itself.
The matrix works on a small scale for a single team. Because it accounts for the organizational side, it also holds up company-wide, as a picture of where a company's many teams stand. It still isn’t a ranking.
A generic model is a starting point. Tune the scales to your context; researchers call these situational maturity models. After that, keep the model as stable as you can. After all, you repeat the assessment to make change visible. Sharpen the scale with every round, and you end up comparing apples and oranges and lose the trend over time, which is the whole point of measuring.
Stable doesn't mean frozen, though. If an organization works in a uniform way, most teams land in the same cell, and you can no longer see the differences. When the model stops giving you insight, adapt it. Ideally, refine within the existing levels. That way the existing levels stay comparable across every round, and the line doesn't break.
In larger companies, the agentic software engineering transition takes years, not months. Over that time, models, tools and ways of working will change several times. What we call Orchestrated today may be routine in two years. So ask yourself regularly whether your maturity model still gives you enough transparency and insight.
A dot in the matrix shows where a team stands. For planning, you need the direction. That is why we ask our teams two things: where they stand today and where they want to be in six to twelve months. Every dot becomes an arrow.
I'll show how to work with these arrows using six example teams. The values are realistic, but they are not the responses of our teams.
Current and planned position in the maturity matrix, example values for six teams A to F
Start with the end points. The plans are balanced: all six target points lie on the diagonal. Organizationally, five teams want to reach the team standard, Adopted. No team plans to rebuild the way it creates value around agents within this period. Team C deliberately wants to stay where it is. That, too, is an honest answer, and it may be exactly right for the team's current situation. This is why we show Agentic delivery in the matrix as the operating model and still say that a high rating is not a goal. Agentic delivery is a direction for certain teams, not a mandatory stop for everyone.
Then look at the direction of the arrows, because it determines the remedy. Team F currently sits in the adoption gap, as a Tech frontrunner. On the technology side it is already at Orchestrated; its arrow points only to the right. It plans no technological leap. What it needs above all is backing, shared standards and a harness the whole team shares. Team D wants to make working with agents its team standard without changing its technology level. The remedy is organizational. Teams A and E also want to move up one technology level. For them, technology and organization have to grow together: a shared harness and skills built across the whole team.
Some of the planned technology jumps, by contrast, are big. Within six to twelve months, Team B wants to go straight from Assisted to Orchestrated, and from Enabled to Adopted on the organizational side. An arrow like that deserves scrutiny. Can the organization keep pace? If the technology runs ahead, the team ends up in shadow automation along the way. And who checks the code when agents suddenly run entire loops?
This question is not limited to the example. In our real AI4SE status overview, we also ask teams what is currently blocking their way. The most frequent answer in the latest survey: the review and test loop.
That follows logically. When coding agents produce code faster than people can read and test it, a bottleneck forms that causes problems in the team's daily work. Either work piles up in review, or review becomes superficial and quality suffers. Fiona Chen and James Stratton of Harvard University analyzed data from more than 700 firms in a working paper and describe the same pattern. Coding agents raise productivity in programming. On output, however, the two find no statistically robust increase. They locate the cause in review: it takes longer, changes need rework more often, and there are more comments.
Our conclusion: software verification is a core requirement for every team's harness. What people can no longer fully read, the machine has to check, on several levels. Autonomous end-to-end validation has agents operate the application the way people use it day to day, and checks the result against acceptance criteria based on business requirements. Mutation testing tests the tests themselves. That matters especially with generated tests, because an agent quickly writes tests that pass and prove nothing. Security tests catch insecure patterns a model brings along from its training data. Architecture tests protect layers and dependencies, so the codebase doesn't erode a little further with every agent run. And in a multi-agent architecture, different LLMs check each other and challenge each other as sparring partners. The author and the reviewer must be different people. Or different models.
Without automated verification in the harness, the loops run, but nobody can vouch for their results. They won't get the backing of team and management that way, and the team stays stuck in the adoption gap, as Shadow automation or Tech frontrunner. Only with verification does the move to the right, to Orchestrating team, succeed.
Where does your team stand if you are honest, perhaps one level lower than it feels? Does the gap point toward the organization or toward tooling?
The matrix shows where a gap is. It doesn't show where it comes from. That is why it can make sense to run a developer experience (DevEx) survey across all engineers alongside the maturity assessment. Its results help teams rate themselves. And they give you a more detailed basis for finding the causes of a tooling gap or adoption gap: is the tool missing, or access to it? Is it time, knowledge or backing? With those answers, you choose the remedy more precisely.
We have shared the matrix here so you can use it for yourself. As an IT services company, we pass on what we are learning from building the model and from guiding our own change. And we are glad to support other IT and product companies in their transition. I look forward to your experiences, objections, and your own versions of the matrix in the comments.