The Biggest Risk in AI-Written Tests: Giving Itself a Perfect Score

The biggest pitfall in AI code testing is when it starts grading its own homework.

The problem in the news is quite typical. An AI assistant writes a function, casually writes the tests too, all tests pass green, the review looks reasonable, so it gets merged. The bigger trouble is that these tests likely only verify its initial assumptions, failing to cover actual user needs. Code and tests share the same source, so errors will too. If a function is misunderstood, the accompanying tests will probably follow suit, resulting in a very respectable false sense of security.

This reminds me of another set of experiments. Students using ChatGPT solved more problems on the spot, reportedly solving 48% more, and those using AI tutoring reached 127%, but students without AI dependency were more stable in subsequent tests. On-the-spot pass rates cannot be equated with real capability. All-green doesn't mean you defined the boundaries correctly.

This is particularly similar to autonomous driving validation. I've been running FSD and Robotaxi for weeks, and recently I've been looking at Tianshu Navigation Autopilot. What I fear most is simulation scenarios generated based on model capabilities. The car runs through, logs are all green, but that doesn't mean edge cases are truly covered. Especially takeovers, long-tail intersections, construction detours, and night-time ghost probes—these cannot rely solely on the model proving itself.

So, AI-generated tests can be used, but roles must be separated. Let it write templates, supplement boundary cases, and organize regression lists—all fine. But what exactly to test, what counts as passing, and which risks cannot be missed—this cannot be left to the same model to decide.

It's best to have a set of independent acceptance criteria plus continuous monitoring. After code goes live, you still need to watch for behavioral drift.

In my tests, AI efficiency gains are real, but it's better suited as an assistant, not a judge. Letting AI write tests saves time. Letting AI prove itself will eventually lead to paying debts in the production environment.

---

📌 This article is compiled from Hacker News. Original link: https://www.michaelbromley.co.uk/blog/the-problem-with-your-ai-tests/

Copyright belongs to the original author. This is a compilation and independent analysis based on public reports.