What the Data Actually Says About AI-Assisted Development

This article is a look at what the research actually says about AI-assisted development, drawn from the numbers I shared in my Stir Trek talk on protecting your codebase from AI mistakes.

I think we had an inflection point mid-year last year in terms of quality, so keep that in mind with some of these scary metrics, especially the early ones - and the most recent telemetry does show some of these numbers moving.

The Promise vs. The Reality of AI Speed

The big reason most folks want to do AI beyond "it's cool" is they want to go fast. They want to be able to write more code. They want to be able to ship things faster. They want to be able to have a faster time to market on things, and that all makes sense.

But there have been some studies done, especially the METR study from early 2025, where they took a team of developers, had them use agentic AI to do their work, and asked how much faster they were going to be. They asked economists for forecasts and they said probably 39% faster. Machine learning experts and data scientists said, "Well, maybe 38% faster." They asked the developers themselves, and they said, "Well, maybe 24%." Then they let them run wild for a sprint or two, collected the metrics, and asked the developers again, "How much faster were you?" And they said, "Well, I think we were about 20% faster." But the metrics at the time showed that they were actually 19% slower. So they had this Dunning-Kruger effect where they thought they were faster, but they were actually slower. I think that if we redid this experiment now the results would be different, but this is what it was at an early 2025 stage.

So what slows us down? Probably the biggest things are quality and trust and the amount of effort required. It takes a lot of time to get good results out of AI agents, most folks are saying, and it takes some time to understand and trust those results. Others say there's a skills component to this as well.

What the Code Quality Data Shows

The GitClear study went in and asked how much faster we are as developers. And it turns out we actually are faster. We're delivering more epics. We're delivering more tasks and we're sending in more pull requests. But they did note that code churn is up significantly: nearly double the pre-AI baseline. They measure that based on the frequency at which individual lines of code are updated within a week or two of initially changing or being added. So we are churning; we're changing our lines of code pretty significantly. That can indicate either that we are continuing to evolve the software, or it can indicate bugs or just general uncertainty or merge conflicts or whatever it might be. So that's a little bit of an alarm bell right there.

More blatant alarm bells would be the number of publicly facing incidents. We're having significantly more incidents per month, up 125.4%, and more bugs per pull request as well: 10.83 issues per AI-authored pull request against 6.45 for human-written ones. These are concrete measurements from last year that folks have found associated with this AI adoption. And another report says we're seeing 1.7 times more total issues, 1.75 times more logic errors, and 1.4 times more critical issues. These are all critical things that are happening in the community adoption.

Security Vulnerabilities Are Spiking

Of the mistakes that happen, we're seeing a lot more security vulnerabilities, particularly in Java code for some reason. I'm not sure why - Veracode has Java failing security tests 72% of the time, against 45% for C#, 43% for JavaScript, and 38% for Python. And 45% of those are going to be OWASP Top 10 vulnerabilities: things that most developers should be aware of, such as SQL injection.

The Maintainability Problem

One of the more concerning areas for me is maintainability and being able to sustain this over time. AI isn't thinking about refactoring the way that a human is: refactoring has fallen from 21% of changed lines in 2022 to 3.8%. It's not thinking about code holistically and long-term the way a human is that's trying to solve a particular problem.

The metrics around code duplication are interesting and surprising to me, because I figured the duplication increase would be larger than 4% (8.3% of changed lines to 12.3%). I believe this is just exact lines of code duplicated, versus duplicating functionality that already exists in other places. I've seen a lot of CSS duplication everywhere. I've seen a lot of similar objects being created because it's not searching your code to see whether this already exists anywhere else. Some of the models do, some of the agent harnesses do, but it's not perfect. It's not as good as an engineer would be in that regard. So it's almost like a legacy code factory at that point.

And there have been some studies on what happens with these vibe coding startups: they start out doing great, and then they start slowing and slowing over time, and then eventually they become future customers of ours. That's because they can no longer understand their own code or sustain that velocity, and they need a rewrite, they need a retooling, whatever it might be. Vibe coding is great for a short period of time. It's not sustainable in general.

Where This Leaves Us

This data tells us what's going wrong, but not much about what to do instead. Some of it is process rather than tooling: Erica Krumlauf, our COO, makes the case that AI doesn't fix broken delivery processes, it exposes them. The rest of this series covers the practices that help, starting with how context impacts what your agent produces and continuing through testing, code review, and CI/CD safeguards. You'll find the rest of the series on the AI-Assisted Development hub on Leading EDJE's blog.

This article is adapted from a talk presented at Stir Trek. Watch the full presentation below: