Three Production Postmortems: Flyway, JAR Deploy, and Duplicate Beans

Database migration discipline is the most boring, most underrated part of backend engineering, until it crashes production. On this portfolio, 189 Flyway migrations (up to V199) govern the PostgreSQL schema of a backend with over 3,100 Java classes. Across three distinct production incidents, each traceable to a violation of the same implicit rule, "never touch something the system already considers written and verified", I learned how thin the margin is between boring discipline and a production boot crash.

Postmortem 1: a comment broke the checksum

Flyway computes a checksum of the full text content of every already applied migration file, and compares it on every startup against the value stored in the flyway_schema_history table. If they do not match, Flyway refuses to start the application. On 2026-06-20 I edited a single SQL comment in an already applied migration (V152), thinking a comment could not have functional effects, to clarify an intent that had become unclear months later. The backend did not restart: FlywayException: Migration checksum mismatch. No column, no table, no logic had changed: only text after a -- double-dash. Flyway does not distinguish between a comment and a statement, it treats the file as a byte blob with a checksum, period.

The operational rule I now apply, with no exceptions: an applied migration is frozen forever, including every comment. If an intent needs clarifying, write a new migration that documents the why in the commit message or in a comment on the new migration, never by editing the old one.

Postmortem 2: the JAR deploy race

Manual backend deploy follows the pattern git pull + mvn package + systemctl restart. Before the fix, the deploy script copied the new JAR over the running one (cp) while the previous JVM was still running and had the file memory-mapped. The observed production result: NoClassDefFoundError on core Spring classes, because the classloader was reading a file partially overwritten mid-way through a non-atomic cp, while the old process was still alive and serving traffic. The symptom is particularly insidious because it is neither a compile error nor an application bug, it is a binary file corrupted at runtime that looks like a missing-dependency problem.

The fix has three parts, and all three are necessary, none alone is enough: stop the service before copying (not after), copy the new JAR with an atomic mv instead of an in-place cp, and reset systemd's failed state if the previous restart went wrong, before restarting again. Skipping any one of the three reproduces the race intermittently, which is the most expensive kind of bug to diagnose because it does not recur on every deploy.

Postmortem 3: two beans, same simple name, boot crash

Two @Service classes in different Java packages but with the same simple name (e.g. ExportService in two separate business verticals) blow up the Spring boot with ConflictingBeanDefinitionException, because Spring, without an explicit name, derives the bean name from the simple class name, not the fully qualified name. The genuinely uncomfortable part: unit tests do not catch this, because no single test loads the full application context with both verticals together, only a full-context integration test reveals it, and in practice that full-context test runs rarely during routine development, often only before deploy or never.

With over 25 separate business verticals added over time, the risk of a simple-name collision grows combinatorially, not linearly, with the number of verticals. The rule I adopted: every new @Service in a specific business vertical gets an explicit, prefixed name (@Service("phExportService")), never left to the default, and at least one full-context integration test runs before every deploy.

The common pattern: immutability as a boundary, not a style

The three postmortems look independent, but they share the same root cause: something the system treats as already written, verified, immutable, gets modified or overwritten at a moment the system does not expect it. An applied migration, a running binary, a simple name Spring assumes is unique. Immutability is not an elegant coding style to follow when convenient, it is a system boundary that, when violated, produces a deterministic crash or an intermittent race, never a gentle warning.

When this discipline is overkill

On a single-person side project with one environment and infrequent manual restarts, the risk of a JAR deploy race is low and the cost of a sophisticated stop-before-copy deploy script is not justified. The real breaking point is production with continuous traffic: there, even a few-second race window becomes an error window visible to users, and the discipline stops being bureaucracy and becomes necessary protection.