03 - Four Production Postmortems: Why Green Tests Are Not Enough

"The tests are green" does not mean "the deploy is safe". It is a sentence I have repeated for years while still violating it in practice, until four distinct production incidents - weeks apart, with completely different causes - forced me to take it seriously. In all four cases the build was green. In all four the problem was invisible to any unit test suite, because it lived exactly at the boundary between "the code compiles and is tested in isolation" and "the system actually boots in production".

Four crashes, all starting from a green build

These are not imaginary incidents, or incidents built to look good in an article. They are four real postmortems of a Spring Boot backend in production, each with its own detail file in the linked repository: symptom, root cause, why the tests did not catch it, the fix and the rule to avoid repeating it.

• Flyway checksum mismatch: removing a single comment from an already-applied migration crashed the boot in production.
• Silently disabled ArchUnit gate: an architecture test written to protect a security boundary had run zero tests for months, while still reporting "build success".
• Race condition in the JAR deploy: an in-place replacement of the executable file under a JVM that was still starting corrupted the class loading.
• Duplicate Spring bean: two classes with the same simple name in different modules brought the boot down with ConflictingBeanDefinitionException.

The four cases touch different levels of the system (the database, the tests, the deploy, the startup of the Spring context) and the order of the list is not a ranking of severity: what they have in common is that the green signal measured something different from what then broke, as the section on the immutability boundary further down shows.

Incident 1: the comment that broke Flyway

Flyway computes the checksum over the entire resolved content of a migration file, not just over the executable SQL statements. A seemingly harmless code cleanup had removed an obsolete comment line from a migration already applied to the production database. Zero lines of SQL touched. At the next boot, Flyway compared the checksum recorded in the schema-history table with that of the file on disk: they no longer matched, and Flyway refused to proceed, making the initialization of the Spring context fail.

No unit test suite ever touches a real schema-history table: the problem was invisible by construction to any test that did not start a complete Spring context against a real database.

Incident 2: the ArchUnit gate that never ran

An ArchUnit test written with the static-field @AnalyzeClasses + @ArchTest pattern depends on a separate JUnit Platform test engine (archunit-junit5) to be actually discovered and executed. With a certain combination of Surefire and JUnit Platform versions, that engine was not included in the set of engines run by Surefire. The test reported BUILD SUCCESS and "0 violations" - but the real report said Tests run: 0. The gate meant to protect a security boundary between application layers had never executed a single assertion, on any commit, since it had been introduced.

This is the most uncomfortable of the four cases: it is not that the tests "failed to catch" the problem in the classic sense, it is that the problem WAS the test itself. A green CI dashboard almost never prompts anyone to check manually how many tests actually ran for a specific class.

Incident 3: the race condition in the JAR deploy

The deploy script installed the new artifact with an in-place copy over the existing JAR, followed by a service restart. If a previous process was still starting at that exact moment (for example an automatic restart after a health check had failed shortly before), the copy overwrote the bytes of the JAR on disk WHILE that JVM was still lazily loading classes from that file. The result: corrupted class loading, with NoClassDefFoundError errors on classes of the framework itself (not of the application code) - a strong diagnostic signal that the JAR file had changed under a live process, not that the build was broken.

Incident 4: two classes with the same name, one conflicting bean

Spring derives the default id of a bean from the simple name of the class, decapitalized, not from the fully qualified name. Two independently developed modules each defined an annotated class with the same simple name in different packages. The component scan at startup found two beans that wanted the same default id, and Spring refused to register both: ConflictingBeanDefinitionException at boot, service down. No unit test would ever have caught it, because unit tests instantiate the class directly and never load a complete Spring context.

What they have in common: the immutability boundary

Looked at together, the four incidents share the same structural root cause, even though the symptom is different each time: an immutability boundary - an already-applied migration file, a binary artifact served by a live process, a bean id that is unique by implicit contract - was violated without any mechanism mechanically preventing it. In every case the definitive fix was not "pay more attention", but a structural constraint: never edit an applied migration, replace an artifact only with stop plus atomic rename, give a bean an explicit name prefixed per module when the collision is plausible.

For the version of these incidents read as the work of a platform (which guardrail replaces which rule, and what it costs) see Three production postmortems. On the dead ArchUnit gate and the frozen baseline that replaced it, see ArchUnit and Clean Architecture.

Is the architecture gate worth the setup cost? An honest trade-off

Incident 2 raises an uncomfortable question that is worth facing head-on, instead of just telling the fix: is an architecture gate like ArchUnit really worth the maintenance cost for every team? My answer, after living with a silently dead gate for months, is: it depends on the size of the code, not on good intentions.

On a small codebase, maintained by a single person, with fewer than a few dozen classes per layer, an automatic architecture gate adds more maintenance cost (versions to keep aligned, one more test engine to check is really executed) than the protection it takes away: personal discipline is enough, and it is cheaper. Beyond that threshold - more contributors, more modules, a boundary that someone MUST be able to violate by mistake without noticing - the gate becomes indispensable, but with a non-negotiable condition: you must verify that it really runs tests, not just that it reports "build success". A gate that reports zero violations because it never runs is strictly worse than no gate, because it generates an active false sense of security instead of a declared absence of security.

October 2026 addendum: six incidents that are not about boot, but about process

The four incidents above were born inside the backend. In the following weeks the same pattern reappeared outside the application code: in scripts, in the reverse proxy configuration, in the routing of the SSR server and in the work of automated agents on the repository. The signal was always green (exit 0, a page answering 200, a successful build) and the signal did not measure the thing that mattered. I add them here, dated, because the original article claimed that unit tests are not enough; these six say that not even a command's exit code is enough.

1. The script that deleted files when validation failed

The article translation script, when the validation of a translation failed, ran f.unlink() on the en.html and en.json files of the article, and printed «validation fail, file rimosso» (the message is in Italian: "file removed"). It deleted 238 files in two rounds and produced, in the same corpus, 225 corrupted English articles out of 1419 (15.9%), with the model's boilerplate and XPHX<n>XPHX placeholders in place of the text. The files came back because they were tracked in git: not thanks to any merit in the script's design. The design flaw is a destructive default: a check that fails must quarantine, not delete. The rule now in force is "it is not used, not even to repair the damage it produced".

2. The process that outlived whoever had launched it

The first time, the cause was attributed to an agent that had followed its own definition instead of the brief. It was a plausible explanation, and it was wrong: the second time it became clear that the deletions came from the script above, launched in the background and left alive after the agent had finished. SIGTERM did not stop it: it needed SIGKILL. The lesson here is about the method of diagnosis: a cause that fits the facts is not a proven cause, and only the second episode disproved it. The operating rule is a two-line check after every round of automated work: look for surviving processes (ps -eo pid,cmd | grep translate-articles) and count the files deleted in the working tree (git status --short | grep -cE '^( D|D )').

3. The PDF conformance gate that could not fail

A conformance check on PDFs trusted the exit code of gs. Ghostscript exits with 0 even when the requested operation fails, so the gate was blind in the same way as the ArchUnit test of incident 2: it reported success without having verified anything. A gate is verified by deliberately making it fail on a known input and checking that it signals; the right check reads the output or the produced file, not the exit status.

4. nginx: add_header adds, it does not replace

curl -I on the favicon returned two conflicting Cache-Control headers: the one from the origin (86400 seconds) and the one set by the location (30 days). Per RFC 9111 two contradictory directives make a malformed response and the recipient may use the first, the second or neither. The remedy is proxy_hide_header Cache-Control; in the location that uses add_header in front of a proxy_pass. Hidden cost: the location has proxy_cache_valid 200 30d, so without a cache purge the old behavior stays served for a month. The defect emerged while investigating another problem, the generic icon in search results, which had a different and earlier cause.

5. app.use with a path matches by prefix

A handler mounted on /blog/:slug also claimed /blog/series/<x>, assigning slug = 'series' and answering 404: all 167 series pages were left out. No test on the single route could see it, because the route worked for what it was meant to serve. Rule: a guard mounted on a parametric path must verify that the slug is the last segment of the path.

6. The "not found" with HTTP 200

The dataset detail pages caught every error of the preview loading with .catch(() => null) and showed an empty table with status 200. Behind it were three overlapping defects: data not loaded, a slug-to-table map frozen at an old catalog (3 datasets resolved out of 41 published) and the swallowed error. For a search engine an empty page with 200 is worse than a 404: it is indexable, duplicate content. The fix separates the states: 404 becomes "not published", network errors, 5xx, 429 and timeouts become "error" with a Retry button, and a 200 with empty columns and rows counts as "not published".

Two more, from the backend: the same class, in July

Two more incidents of the same kind, in the relationship between the backend and the ml-service, are told in the series on service architecture, in the article on communication patterns: a circuit breaker shared among about thirty endpoints and a timeout that was configured but never applied.

The cost of these rules. The check after every round of agents takes a few seconds; a verification that deliberately makes a gate fail requires a test case for each gate; the distinction between the states of a page costs one more component and a test suite. It is the right price when a silent error can delete 238 files or make 41 pages non-indexable, and it is a luxury when the tool is throwaway and run only once. The criterion is the same as in the section on the architecture gate: where an error leaves no trace, the check must leave one itself.

What I would change today, with hindsight

If I had to start from zero, the one thing I would change for all four incidents together is the same: add, for every mechanism that depends on a state invisible to the compiler (a schema-history, a file served by a live process, a runtime component scan), an explicit check run at least once before the deploy, not just "the code compiles and is tested in isolation". It is not generic advice: it is the specific lesson, repeated four times with different symptoms, that the compiler and the unit tests protect a smaller subset of the system than it seems.

In the "Original Research" series, previous: 02 - GEO Retrofit Benchmark; next: 04 - INP and Core Web Vitals: from 59 to 88.