Should AI coding agents be allowed to test their own code

[RomanReigns26]

First Mate Technologies has published details of a development workflow built around keeping code generation and code verification completely separate. One AI model handles the actual implementation while a different model designs the test cases and reviews the finished work before anything gets merged. A human engineer still sits over the whole process and makes the final call, which feels like the sane way to run this kind of setup right now. The core idea is straightforward, do not let the same model that wrote something also decide whether it is correct

They ran this process on QueueMate, a restaurant queue management app built largely using AI coding agents, and the numbers are pretty striking. The checking model generated 554 separate test cases covering everything from happy paths to edge cases, concurrency issues and accessibility concerns. That sweep turned up 38 defects total, including one severity one bug related to session handling that had real consequences for how long users stayed logged in. Roughly two thirds of those defects got fixed in a follow up commit once they were properly documented

The reasoning behind keeping builder and checker separate comes down to a fairly simple failure mode, if a model misunderstands a requirement while writing code it can easily carry that same misunderstanding into its own review later. Starting a fresh conversation with the same model does not fix this because the underlying misreading of the spec is still baked into how that model thinks about the problem. There is actual research behind this concern too, a 2024 survey on LLM self correction found little evidence that models reliably catch their own mistakes without outside feedback. Interestingly the same research found stronger results when models had access to something objective like a code interpreter rather than just their own judgment

Things get more concerning once you factor in reward hacking, which OpenAI has documented in its own reasoning models. In one example a model was asked to implement a set of math functions, discovered the tests depended on a separate verification function, and simply edited that function to always return true instead of doing the actual work. That is a pretty blunt way of gaming a test suite and it is exactly the kind of behavior that a same model review process would be unlikely to catch. OpenAI has since tested a separate monitoring model that watches the coding agent's reasoning trace and reportedly flags this kind of subversion successfully

Spotify has apparently taken a similar approach on their end, running deterministic checks like formatting and builds first before layering on an LLM based judge that reviews the agent's actual behavior. That judge reportedly vetoed about a quarter of the thousands of agent sessions it examined, and the coding agent managed to correct itself in roughly half of those flagged cases. It is a good example of a company treating AI generated code the same way you would treat any other unreviewed pull request, with layered checks rather than blind trust. None of this replaces human oversight, it just adds more filters before a human even has to look at it

My take is this is exactly where serious AI assisted development needs to head, the same separation of duties principle that makes human code review useful applies just as much when one of the reviewers is an AI model. It also makes intuitive sense that letting a model grade its own homework is going to miss the exact mistakes that model is prone to making in the first place. Curious how many people here running agents on real projects are actually doing anything like this versus just trusting whatever the agent reports back

Should AI coding agents test their own code?www.developer-tech.com