Harvard study finds AI coding agents boost code output by 30% but not finished software

[TransformerCipher]

Ars Technica has a write up of a new study that puts real numbers on AI coding agents. The paper is called Artificial Intelligence in the Firm: Bottlenecks in Software Production, by Harvard economists Fiona Chen and James Stratton. It uses data from Jellyfish, an engineering analytics platform. That covers about 300 million work events from GitHub, Jira and Google Calendar, from January 2021 to March 2026, across 725,938 workers at 718 firms

The headline finding is that agents do make developers produce more. Lines of code per worker per month went up by about 30%, from a baseline of 5,017. Commits rose about 20% and pull requests about 23%. On the face of it, that sounds like a big productivity boost

The catch is in what happens next. Jira issues resolved, which is a decent proxy for finished work, showed no significant change, and the authors rule out gains above 12%. Code review time went up by 49%, an extra 3.45 days on a baseline of 7.03. The share of pull requests sent back for changes nearly doubled, from 0.13 to 0.25, and review comments per pull request rose 35%

In the authors' words, productivity gains do not fully pass through to changes in software output or employment. Their conclusion is blunt: code review becomes the bottleneck. More code is being written, but humans still have to check it, and that is where things pile up. The study also found no significant effect on overall or engineering employment, and simpler AI assistants showed smaller, mostly non significant gains

This matches what a lot of developers say anecdotally. Writing code was never the slowest part of shipping software. Understanding requirements, reviewing changes and testing take most of the time. If agents flood the review queue, teams need to rethink how review works, possibly with AI helping there too

Does this match your experience with AI coding tools? And is code review the next thing that needs automating?

AI coding agents generate more code, but not more softwarearstechnica.com