06The QMS as backbone

In regulated software, the evidence has always cost as much as the code. AI changes the cost of the evidence too, provided the quality system is built so that the AI can work inside it.

Anyone who has built software for a regulated industry knows the pattern. The code is half the work. The other half is the evidence: requirements traced to tests, risks traced to the controls that address them, decisions recorded with their reasons, every dependency justified, every deviation explained. In medical devices, standards such as ISO 14971 for risk management and IEC 62304 for the software life cycle define it; railways, aviation, automotive and finance ask for the same kind of evidence under other names. It is usually produced late, reconstructed for the audit by people who were not there when the decisions were taken. It is expensive, unpopular, and often fiction.

With an AI, the evidence can be produced with the work, by the same process that does it. That takes a practice Scrum already has, a Definition of Done that every piece of work must clear, made specific as in Beyond the Demo, and a quality system built so that the AI can work inside it.

Where does the quality system live?

I made one choice at the start that shaped everything else: the quality management system, or QMS, would live next to the code, as plain text files under version control. Each requirement, constraint, risk, decision and open item has an identifier, sits in a Markdown file, and is changed the way code is changed: in a commit, with a message, reviewable line by line.

A QMS in that form is one an AI can read and write. Claude can open the constraints register before writing a story, check the risk register before closing one, and add an open item the moment it finds a gap, all in the same commit as the code. The evidence is then written while the work happens, by the same process that does the work.

What the QMS holds
  • a Definition of Done every story must clear, covering function, code quality, tests, safety and traceability, security and supply chain, data protection, and documentation;
  • a register of design constraints, the rules every component must respect;
  • a risk register with about thirty hazards, and the risk management file that governs it;
  • a register of third-party software, each library listed with its version, purpose and licence;
  • architecture decision records;
  • about a hundred open items: every stub, deferred decision and known gap, dated and cross-referenced;
  • the regulatory file: intended use, classification rationale, software life-cycle file, cybersecurity file, clinical evaluation and post-market surveillance plans.

Two of these held on paper only. The register of third-party software lists the components the product asks for by name, and misses six of the eleven it ships. The Definition of Done forbids an unresolved high vulnerability, yet since 30 September no build has produced the bill of materials that check reads, and a driver with three high advisories shipped in the meantime. Chapter 07 counts both among the slips; Appendix E has the numbers.

Its rule is that every constraint is referenced somewhere, every requirement traces to the tests that verify it, and anything marked as a draft or a stub has a matching open item. The review findings deferred with a reason, about 245 of them (Figure 5), are kept in a log of their own; when work on the product resumes, they will be at the very top of the to-do list.

To set it up, I designed a plugin for BMAD and built it with Claude. It asks a few questions, such as the device's risk class, the markets it targets and whether it handles personal health data, creates the files the regulations require for that case, and wires the QMS into BMAD's workflows so that they load the relevant files before they start. The wiring is the part that went wrong, and silently; chapter 07 tells that story. The plugin is published under an open licence, with its tests, at github.com/jouleco-fr/bmad-samd-qms, and Appendix F opens it.

What did AI change?

Traceability, retrofitted

In mid-September an audit found that the convention for linking each test to the requirement or risk it verifies had never been adopted. In a traditional team, retrofitting traceability to well over a thousand existing tests is the kind of task that is planned, postponed and never done. Here it took two weeks: the tests now carry more than 1,200 traceability annotations, and a script generates the traceability matrix from the code itself, including its own gaps, such as the requirements no test traces yet. Nobody maintains it by hand. A trace says which test covers a requirement. Whether that test would notice a mistake is another question, and Appendix B puts it to the safety-relevant code.

Open items written as they arise

Every gap Claude finds while working is logged with its date, its cause, what it affects and what was decided. Some entries read like small incident reports, where a human team under deadline pressure writes "TODO". The register made it possible to see, later, what had slipped.

Rules turned into build checks

By the end of September, the rules that mattered most were checked on every build: test coverage that can only rise, at least 95% branch coverage on code that controls a risk, and no test without a link to a requirement. Claude wrote those checks too.

The traceability chainRequirement, risk, story, code, test and Definition of Done, each linked to the next by an identifier. Three rules are checked on every build; the traceability matrix is generated from the code.Requirementrequirement idRiskrisk idStorystory keyCodesafety classTest@Traces(…)Definitionof Doneevery storyCHECKED ON EVERY BUILDNo test without a link to a requirement or a riskAt least 95% branch coverage on code that controls a riskCoverage that can only riseThe traceability matrix is generated from the code, and lists its own gaps.
Figure 6. The traceability chain: requirement, risk, story, code, test, Definition of Done, each linked to the next by an identifier the build can check.

What must a person own?

Some open items cannot be closed by any amount of engineering: sign-off by a regulatory affairs specialist, a meeting with the regulator before submission, management's formal acceptance of the risk thresholds, the clinical judgement behind the hazard analysis, the surveillance of a product that has real users. Claude drafts what can be drafted. It does not sign. Each of these items names the person or external role it needs, and none is skipped without notice. That line, between what an AI can produce and what a person must own, is one of the most useful things a quality system can make explicit, and it is the line a leader will be asked about first.

For your own project

Few readers will build a medical device, but most work under some standard: an industry regulation, a security certification, a customer's audit, or an internal rule that every significant decision is recorded.

  • Keep the quality records where the AI can read and write them: plain text, next to the code, under version control, with an identifier for every requirement, risk, decision and gap.
  • Let the AI do the work that is expensive and dull for people: retrofitting links, logging gaps, regenerating matrices, writing the checks. Keep the decisions and the signatures for people.
  • Make the quality system part of the definition of done for every piece of work, from the first story.

All of this assumes the AI actually reads the rules. For ten weeks, mine did not.