Appendix BDo the AI's tests catch anything?
The same AI wrote the code and the tests, and the tests pass. Whether they would notice a mistake is a different question, and mutation testing answers it. A defect that slipped past every test shows where its answer stops.
Status, 2 November 2026
This appendix is a snapshot of work in progress. Done: the runs reported below, the fix of the pattern they led to, a build gate that refuses it, and the two test gaps closed. Open: a person's review of the survivors (OPEN-113), the test-strength rule, and the improvements ranked at the end. Their outcome will be reported in a later version of this appendix. Numbers in the form OPEN-nnn are entries in the project's register of open items, part of its quality system (chapter 06).
A sceptical engineer reading the article will ask this before anything else. Claude wrote about 24,000 lines of product code and 31,000 lines of tests, and the tests are green. But a test can run a method, check nothing, and still count towards coverage. The device's build refuses any change that lets the branch coverage of its safety-relevant classes fall below 95%, and that figure says only that the code was run.
Mutation testing goes further. A tool plants one small mistake in the compiled code: it flips a condition, empties a return value, deletes a call. Then it runs the tests. If one fails, the mistake was caught and the mutant is "killed". If they all pass, the mutant "survived": the tests would have let that mistake through too.
What was measured?
I had Claude run PIT 1.30.1, a mutation-testing tool for Java, with its JUnit 5 plugin 1.2.3. The first run, on 29 October, took only the classes the device's build lists as safety-relevant: the filter that decides whether a request may act on a page, the caps written into the code, the declarations of what each page accepts, the client and caches for permissions, and the renderer that decides what appears on a page. A reviewer then made two fair points: the scope looked chosen, and PIT's default set of mutators is the mildest in common use. So the runs were repeated on the whole device and on the whole configuration tool, with the default set and with PIT's stronger one.
The score counts every mutant. The strength counts only the mutants in code the tests actually run.
| Scope | Mutators | Mutants | Survived | Not run by any test | Score | Strength |
|---|---|---|---|---|---|---|
| Device, safety-relevant classes | default | 299 | 10 | 15 | 91.6% | 96.5% |
| Device, safety-relevant classes | stronger | 451 | 39 | 23 | 86.3% | 90.9% |
| Device, whole module | default | 1,038 | 41 | 163 | 80.3% | 95.3% |
| Device, whole module | stronger | 1,331 | 87 | 206 | 78.0% | 92.3% |
| Configuration tool, safety-relevant classes | default | 300 | 37 | 14 | 83.0% | 87.1% |
| Configuration tool, safety-relevant classes | stronger | 425 | 49 | 21 | 83.5% | 87.9% |
| Configuration tool, whole module | default | 2,471 | 288 | 331 | 74.9% | 86.5% |
| Configuration tool, whole module | stronger | 3,290 | 430 | 407 | 74.6% | 85.1% |
Three things stand out. Where the device's tests run, they are about as strong across the whole module as in its safety-relevant classes, so the first scope was not flattering them; what the whole module adds is code no unit test runs, mostly the servlets that build the pages, which only live checks exercise. The stronger set takes about five points off. And the configuration tool's tests are clearly weaker than the device's, 85 to 88% strength against 91 to 96%. With no hand-written baseline for this code, none of these figures is good or bad by comparison. What they allow is reading the survivors where it matters. A whole run takes 16 to 20 minutes on the device and up to two hours on the configuration tool.
What survived where it matters?
In the safety-relevant classes, the stronger set left 39 survivors in the device and 49 in the configuration tool. Claude read all 88, tracing each one to its line of code.
| What the surviving mistake would change | Survivors |
|---|---|
| Wording: log messages, page text, headers, the content type of a refusal | 36 |
| How a page is displayed, in the part of the renderer the build does not list | 10 |
| Nothing, or another check refuses the request anyway | 17 |
| Retry timing and caching: slower, and just as safe | 9 |
| A refusal where there should be none | 6 |
| An administrator's change to their own access takes effect within five minutes, as designed when a refresh fails, instead of at once | 6 |
| A security token stays on a server thread until the next request replaces it | 1 |
| A getter checked only ever on an empty set | 2 |
| A request the page refuses gets through | 1 |
The last one matters. The device's request filter checks a submission twice: may this person open the page, then may they run the action. No test sent a request that the page refuses and the action allows, so a mistake that let the second answer overwrite the first would have passed every test, and let the request through. The code was right; nothing would have noticed if it went wrong.
The getter is a correction to the first version of this appendix, which called it dead code. No product code calls it, but the tests do, and only ever on an empty set.
Claude sorted the survivors twice: once for this appendix, and again to prepare the checklist for reviewing them. The second pass moved one survivor from "nothing" to "wording". No person has checked the sorting yet; that review is recorded as an open item, OPEN-113. A sorting that shifts between two passes is the reason it needs one.
Why did one class score 48%?
In the first run, the device's permissions client scored 48%. It reads its settings from the process environment, which a running test cannot change. So its tests start a second Java process with the right settings, call the client there and check the answer. The tests are real and they pass, but PIT plants its mistakes only in the process it controls, and never sees them run. Thirteen of that run's fifteen uncovered mutants are in code tested this way. Of the other two, one is the real lookup of the action catalogue, which every test replaces with a stand-in. Leaving the thirteen aside, the tests kill about 96% of the mutants PIT can measure, the strength in the table. The score keeps them in.
The defect that every test missed
On 17 September I clicked Save on the page composer and nothing happened. The server log had the answer within minutes. To learn the identifier of a row it had just inserted, the code asked the database driver for the generated keys with a bare flag, then took the first column of the reply. With that flag, the PostgreSQL driver returns every column of the new row, in the order the table stores them. The first column is the identifier only if the table happens to put it first.
The test database, built from the schema migration, put it first in every table. The live tables had grown column by column over many stories, and in the table of page cells the identifier was eighth. The first column was the cell's width. So every new cell was saved as if its identifier were 1, the database refused the reference, and the whole save rolled back. Nothing was corrupted. Every integration test had passed, because every one ran against tables where the coincidence held.
Mutation testing would not have found this either. A mutant changes the code; here the code was unchanged, and the fault lay in the gap between the test environment and the live one. The fix named the identifier column explicitly, in the six places the configuration tool used the flag. The story recorded a lesson with it: search for the pattern before converting anything else. No test was added that reproduces the live column order; the fix was judged correct from the driver's specification, so the test database still differs from the live one.
The lesson that did not stick
That search, run while preparing this appendix, found the bare flag still in fourteen places: nine in the device, two in access control, and three in the configuration tool itself, in the code that deploys a configuration to the device. Those three had been written two days before the fix, and the September audit, which looked only at the code being converted, missed them. All fourteen insert into tables created with the identifier first, and later changes only added columns at the end or dropped others, so the identifier is still first and nothing is broken today. But the pattern is there, and the device's architecture document still prescribes it, in its table of conventions. The lesson lived in one story; the document the agent reads before writing code was never corrected. That is the drift chapter 07 of the main article describes: a rule written down, never turned into a check.
What was done about it?
This time the finding was acted on before publishing, in a hotfix story run through the same process as any other. On 2 November, every one of the fourteen places, and two more in a test, named the key column. A new gate in the build refuses the bare flag in any of the three products. The architecture document now rules the flag out and says why. Both test gaps were closed: a test sends the request the page refuses and the action allows, and the getter is checked on a full set. All 1,646 tests of the three products passed, every gate held, and the build was deployed and checked live the same day.
What did I learn?
About Claude and BMAD, three facts. The tests catch most planted mistakes, and in the safety-relevant code all but one of the survivors were harmless. BMAD's development checklist asks for tests on all the core behaviour a story changes, and the Definition of Done holds the safety-relevant classes to their coverage threshold, so the tests grew with the code. The checks were cheap: the first run and its sorting took one working session, and the fix, with its gate and its tests, one more. And the defect that reached live use was found by a person, by clicking Save.
About the method, each finding points at how the work was done:
| Finding | What it says about the method |
|---|---|
| The bare flag survived in fourteen places, and the architecture prescribed it | A lesson had no path to becoming a check, or into the documents the agent reads; this one now has both |
| No test sent a request the page refuses and the action allows | The tests check each rule on its own more thoroughly than the rules together |
| The configuration tool's tests are weaker than the device's | Test strength was never compared between modules, so the gap went unseen |
| Most survivors sit on wording and on edges: a trailing slash, a truncation boundary | The tests cover the main behaviour better than boundaries and exact outputs |
| The escaped defect came from the gap between the test database and the live one | The test strategy assumed the two matched, and nothing checked it |
| One class sits out of PIT's reach, because it reads its settings from the process | Testability was not a rule of the architecture |
| Mutation testing was run by hand, for this appendix | The strength of the tests was never measured, so it could not be improved |
What do the survivors call for?
No test suite can be shown to be complete, and this one confirms it. Claude writes tests for every story, and the build holds the safety-relevant classes to 95% branch coverage; PIT still found gaps, such as two access rules that no test exercised together. Mutation testing leaves that question open, and gives it a measure that can be watched and raised.
The answer is a rule, run like any other quality rule: defined, measured, verified, and its deviations analysed and corrected.
- Defined: a minimum strength for the safety-relevant classes of each module, set the way the build already sets coverage. The strength measured at the start of November is the floor, about 91% for the device and 88% for the configuration tool with the stronger set; the build refuses any fall, and the floor rises as stories raise the strength. The configuration tool is to be brought to the device's level, since it is treated as part of the device until the notified body decides.
- Measured: PIT runs on every story that touches those classes.
- Verified: a person reviews every survivor in them. For this check, that review is still to be done, and it is recorded as OPEN-113.
- Corrected: each survivor that matters is closed by a test, or recorded as an open item with its reason.
Improving the tests this way is the next turn of the cycle, and the rule is among the decisions waiting at the gate.
What would I change?
A quality system closes a check with an action, and the action starts the next turn of the cycle: plan, do, check, act. This appendix was a check. The corrections are done; the improvements to the method below change how every later story is built and checked. They are proposals, ranked by the benefit I expect, each with the measure a later check would read.
| Rank | Improvement | Why at this rank | Measure |
|---|---|---|---|
| 1 | Every lesson becomes a build check or a recorded reason why it cannot, and the documents the agent reads are corrected with it | It covers every kind of finding, and removes the cause seen here and in chapter 07. Applied once, for this pattern; as a rule, still a proposal | Lessons turned into checks, against lessons recorded |
| 2 | Before each deployment, the live schema is compared with the tested one, and any difference stops the deployment | It targets the only defect that reached live use | Differences caught before deployment |
| 3 | Code review runs PIT on the safety-relevant classes a story touches, and reads the survivors | Weak tests are caught story by story, before they pile up; the configuration tool first, and only the classes touched, since a whole run takes up to two hours | Survivors per story |
| 4 | The build refuses any fall in the mutation score of those classes | It keeps the gains of rank 3, and turns test strength into a trend | The score over time |
| 5 | The test guidelines ask for assertions on boundaries, on exact safety-relevant output, and on rules in combination | Cheap, but an instruction holds only most of the time (chapter 07), and rank 3 checks the same thing | Survivors on boundaries and combinations |
| 6 | Safety-relevant code is handed its settings, and never reads them from the process | Sound, and it affects one class today | Mutants out of reach |
Where does this leave the project?
At a gate. Behind it are a working skeleton, the article's evidence that the approach holds, and checks like this one, which test what the article claims and lead to action. Ahead are decisions, these six among them, and none is taken yet. A later article will say which were taken and what they changed. Until then, the figures above describe the tests as they stood on 19 October 2026, before the fix of 2 November.