Green Tests, Broken Features: The AI Testing Trap

The Trap Nobody Talks About

There's a specific kind of frustration that hits when you've shipped something, watched the CI pipeline go fully green, mentally filed the feature as done — and then someone actually uses it. And it breaks. Not in some obscure edge case. In the main flow you thought you'd covered.

That feeling is getting more common. And AI coding assistants are a big reason why.

Here's what happens. You describe a feature. The assistant writes the implementation. Then, almost as a bonus, it writes the tests too. The whole package arrives together: code, tests, and a confident summary explaining why everything works. You run the suite. Green. You move on.

The problem isn't the assistant being malicious or even careless. The problem is subtler. When the same system that writes the code also writes the tests, those tests tend to validate the code's assumptions rather than challenge them. They prove the implementation is internally consistent. That's not the same as proving it actually does what you need.

What 'Passing' Actually Means

Think about what a test can tell you. At minimum, it tells you the code ran without crashing. At most, it tells you the code produced the specific output you asked it to verify, under the specific conditions you thought to set up.

The gap between those two things is enormous. And most of the ways AI-generated tests fail you live somewhere in that gap.

The most common version: a test checks that a function returns something, not that it returns the right thing. The assertion is assert result is not None when it should be assert result == expected_value. The test passes. The function returns garbage. Nobody knows until a real user hits that path.

A close second is the over-mocked integration. You're testing a payment flow, say, and the mock returns a successful response every time — regardless of what input you give it. The real payment API would reject half those inputs. Your tests say healthy. Your users say otherwise.

And then there's the broad assertion. The condition is technically true for both the correct implementation and several broken ones. An AI assistant writing tests optimistically tends to write assertions that match what it just built, not assertions that would catch the ways the build could go wrong. There's a meaningful difference between those two things.

Why AI Makes This Harder to Catch

Manual test-writing has its own problems, but one thing it does force is a small moment of separation. You write the code, then you sit down and think: what could go wrong here? What does 'correct' actually look like for this function? That friction is annoying, but it's also where good test design lives.

When an assistant produces everything at once, that moment disappears. The output looks complete. The explanation is coherent. The tests are syntactically valid and they pass. There's no obvious signal that anything is missing, because the thing that's missing is a question nobody asked: could this test have caught the actual failure?

That's the right question. Not 'did the tests pass?' but 'if the thing I care about had broken, would any of these tests have noticed?'

Answering that requires looking at tests differently than most developers are used to. You're not checking whether the test runs. You're checking whether the test is capable of failing in the ways that matter.

Code Factory: One Developer's Answer

A tool called Code Factory — built by a developer who kept running into exactly this problem — tries to make that question answerable in practice. It's open source, free, and available via pip as factoryline-code-factory.

The entry point is deliberately low-ceremony. You give it a plain-English description of what you're building:

factory mvp 'Build an approval tracker' --root .
factory studio --root .\my-mvp

That scaffolds a local, inspectable starting point — not a finished product, and explicitly not something it declares ready for production. The distinction matters. The tool doesn't try to be autonomous. It doesn't decide it's done. It creates something you can look at.

The more interesting part is what happens around verification. Code Factory keeps surfacing a set of questions that most AI-assisted workflows quietly skip: What requirement is this change trying to satisfy? Which checks actually support that claim? What evidence is missing or hasn't been updated recently? What should be verified next?

Those aren't complicated questions. But they're the ones that get dropped when everything looks green and you're ready to move on.

The Idea of Stress-Testing Your Controls

One concept in Code Factory's design deserves more attention than it usually gets in testing discussions: actively trying to break your own verification.

The logic goes like this. If a test is supposed to catch a specific failure, you should be able to verify that by introducing that failure and watching the test catch it. Delete the control. Invert the condition. Corrupt the expected input. If the evaluator still passes after you've done that, the test wasn't protecting you — it was decorating you.

This is called mutation testing in formal software quality circles, and it's been around for decades. But it rarely gets applied to AI-generated test suites, where the instinct is to trust the output rather than interrogate it. Code Factory's approach of separating the 'worker's claim' from the 'evidence used to check it' — what it calls the Verifier Plane — is a structural way of forcing that interrogation rather than leaving it to willpower.

Most developers, if they're honest, don't routinely try to break their own tests. It feels redundant when everything's green. That's exactly the moment when it's most worth doing.

The Human-in-the-Loop Problem

There's a broader conversation happening about how much autonomy AI development tools should have. Code Factory takes a clear position: the human decision stays in the loop, always, but that decision should come with a real paper trail.

What changed. What was checked. What was challenged. What's still an assumption.

That last one is the one that usually goes undocumented. Assumptions are everywhere in software — about input formats, about downstream service behavior, about what 'valid' means for a given field. When an AI assistant writes code, it makes those assumptions implicitly. When it writes the tests, it often validates those same assumptions rather than probing them. The result is a codebase that's internally consistent but fragile at its edges.

An evidence trail doesn't eliminate assumptions. But it makes them visible, which means they can be revisited when requirements change or when something breaks in production and you're trying to figure out why.

What the Tool Doesn't Do (and Why That Matters)

Code Factory is local-first. It doesn't reach out to external services, doesn't look for credentials, doesn't deploy anything, and doesn't approve its own work. For developers working on codebases with any sensitivity — which is most codebases — that's not a minor detail. It's the difference between a tool you can actually use on real work and one you have to sandbox.

The open-source licensing (MIT or Apache-2.0) reinforces this. You can read exactly what it does. You can run it on your own machine with your own code and not wonder what's being sent where.

There's also something worth noting about what it doesn't promise. It doesn't claim to make AI-generated code safe. It doesn't guarantee your tests are sufficient. What it does is make the gaps more visible and give you a structured way to ask whether your verification is actually doing the work you think it is.

A Different Way to Think About Test Quality

The shift Code Factory is pushing for isn't really about tooling. It's about how developers evaluate confidence in their own work.

Green tests have always been a proxy for correctness, not proof of it. That was true before AI assistants. But when a developer wrote the tests themselves, they at least had to think through what they were checking. The test was a record of their reasoning, even if imperfect.

When an assistant writes the tests, that reasoning is implicit and often optimistic. The tests reflect what the code does, not what the code should do. Catching that difference requires deliberately stepping back from the output and asking harder questions — questions that feel unnecessary when everything looks finished.

Honestly, the biggest adoption challenge for any tool in this space isn't technical. It's cultural. Developers are rewarded for shipping. Green tests feel like permission to ship. Adding a verification step that might surface new doubts runs against that incentive. Teams that want to use an evidence-trail approach seriously will need to make it part of how they define 'done,' not an optional extra that gets skipped when there's deadline pressure.

That's a harder problem than any tool can solve on its own. But tools that make the right questions easier to ask are a reasonable place to start.

If you've ever shipped a green build and then watched it fail in ways the tests should have caught — that's the feeling this whole approach is designed for. The question isn't whether your tests pass. It's whether they could have stopped the thing that actually broke.