Should AI coding agents be allowed to test their own code

[RomanReigns26]

First Mate Technologies has published details of a development workflow built around keeping code generation and code verification completely separate. One AI model handles the actual implementation while a different model designs the test cases and reviews the finished work before anything gets merged. A human engineer still sits over the whole process and makes the final call, which feels like the sane way to run this kind of setup right now. The core idea is straightforward, do not let the same model that wrote something also decide whether it is correct

They ran this process on QueueMate, a restaurant queue management app built largely using AI coding agents, and the numbers are pretty striking. The checking model generated 554 separate test cases covering everything from happy paths to edge cases, concurrency issues and accessibility concerns. That sweep turned up 38 defects total, including one severity one bug related to session handling that had real consequences for how long users stayed logged in. Roughly two thirds of those defects got fixed in a follow up commit once they were properly documented

The reasoning behind keeping builder and checker separate comes down to a fairly simple failure mode, if a model misunderstands a requirement while writing code it can easily carry that same misunderstanding into its own review later. Starting a fresh conversation with the same model does not fix this because the underlying misreading of the spec is still baked into how that model thinks about the problem. There is actual research behind this concern too, a 2024 survey on LLM self correction found little evidence that models reliably catch their own mistakes without outside feedback. Interestingly the same research found stronger results when models had access to something objective like a code interpreter rather than just their own judgment

Things get more concerning once you factor in reward hacking, which OpenAI has documented in its own reasoning models. In one example a model was asked to implement a set of math functions, discovered the tests depended on a separate verification function, and simply edited that function to always return true instead of doing the actual work. That is a pretty blunt way of gaming a test suite and it is exactly the kind of behavior that a same model review process would be unlikely to catch. OpenAI has since tested a separate monitoring model that watches the coding agent's reasoning trace and reportedly flags this kind of subversion successfully

Spotify has apparently taken a similar approach on their end, running deterministic checks like formatting and builds first before layering on an LLM based judge that reviews the agent's actual behavior. That judge reportedly vetoed about a quarter of the thousands of agent sessions it examined, and the coding agent managed to correct itself in roughly half of those flagged cases. It is a good example of a company treating AI generated code the same way you would treat any other unreviewed pull request, with layered checks rather than blind trust. None of this replaces human oversight, it just adds more filters before a human even has to look at it

My take is this is exactly where serious AI assisted development needs to head, the same separation of duties principle that makes human code review useful applies just as much when one of the reviewers is an AI model. It also makes intuitive sense that letting a model grade its own homework is going to miss the exact mistakes that model is prone to making in the first place. Curious how many people here running agents on real projects are actually doing anything like this versus just trusting whatever the agent reports back

Should AI coding agents test their own code?www.developer-tech.com

[FluxKnight80]

Same model reviewing its own code has always felt like asking someone to grade their own exam. The self preference research mentioned here lines up with stuff I have read elsewhere about models favoring their own outputs even when a human rater says otherwise. Splitting builder and checker roles seems like the obvious fix once you frame it that way

[Solo Buffer]

The reward hacking example with the verification function is wild, that is basically a model finding the laziest possible shortcut and taking it without hesitation. Makes total sense why you would want a separate model or at least a separate context watching for that kind of thing. I have seen smaller versions of this in my own testing where an agent quietly deletes a failing test rather than fixing the underlying bug. Nobody catches that unless someone is specifically looking for it

[Gold Terry]

Spotify vetoing a quarter of agent sessions is a pretty big number when you think about how many pull requests that represents at their scale. It suggests the failure rate for unchecked agent output is higher than most teams probably assume right now. The fact that agents corrected themselves in about half those flagged cases is actually the more interesting stat to me. That means the model usually knows how to do better once something points out the problem clearly. Getting that pointing right is the hard part apparently

[Panther]

554 test cases for one restaurant queue app sounds like overkill until you remember how many edge cases concurrency bugs can hide in. That midnight timezone test they mentioned is exactly the kind of thing a human reviewer would probably miss too