While re-platforming ARES — the 50-module damage-management system I wrote about in the zero-downtime migration post — we kept hitting the same wall, over and over:
you can’t test a damage file you can’t create.
The problem: state is expensive
In a damage-management platform, nothing interesting happens on an empty database. A single meaningful test case is a deep object graph: a claim file with a vehicle, a customer, a policy with entitlements, assigned tasks in the right states, mandatory documents, expertise lines for parts and labor, maybe a spare-parts order halfway through its supply process.
Building that state by hand through the UI takes the better part of an hour — click by click, screen by screen. And whatever you build is subtly different from what your teammate built yesterday, so “it works on my file” became our version of “it works on my machine”.
Unit tests don’t fill this gap. We had hundreds of Pest test files for the pieces. But the bugs that hurt were between the pieces — a task state that blocks a document rule, an entitlement calculation that only breaks after a rejection-revision cycle. Those live in fully-formed files.
And we had a deadline-shaped reason to care: the migration. Every table family we flipped to the new schema needed the answer to one question — does a file behave the same in v2 as it did in v1? Answering that requires the same file, twice, in two systems.
What we built
The Scenario Runner is an internal admin tool that fabricates complete, end-to-end test files on demand:
- Presets. A scenario is a named recipe: “insurance file with unsourced parts”, “mini-repair file awaiting approval”, “file ready for delivery”. Pick one, click run, get a fully-formed file — with every entity in the graph created and wired.
- A v1/v2 toggle. The same scenario can be materialized against the legacy path or the new one. That single switch is what turned a convenience tool into migration infrastructure.
- Legacy Radar. Alongside the created file, the tool surfaces which legacy surfaces the scenario still touches — so “is this flow really v2-native yet?” stopped being a matter of opinion.
One design decision mattered more than the rest: scenarios are built through the same domain services production uses, not through raw inserts. A file fabricated by bypassing the business logic tells you nothing — worse, it can be a state the real system could never produce. Building through the front door keeps every generated file honest, and as a side effect it exercises the write paths themselves.
How it became the parity gate
Once the v1/v2 toggle existed, a workflow crystallized around it. Before a table family flipped to v2-primary, the same scenarios ran on both sides and the results were compared — statuses, computed amounts, timeline events, generated documents. Differences were either bugs to fix or intentional changes to document. Only when the scenario suite agreed did the flip proceed.
That’s what I mean by parity gate: not a dashboard, not a meeting — a tool-enforced checkpoint that made “v2 behaves the same” a testable claim instead of a hopeful one.
It paid off beyond the migration, too. Bug reports now come with a scenario name instead of a fragile list of reproduction steps. New teammates explore the domain by generating files and poking at them. QA stopped hoarding hand-crafted test files like family heirlooms.
What I’d tell you if you’re building one
- Build through your domain services, not your tables. Fabricated state must be reachable state.
- Make scenarios named and shareable. The value isn’t creating a file — it’s two people talking about the same file.
- If you’re migrating, make the target switchable. The moment a scenario can run against both systems, you have a parity instrument for free.
- Surface what the scenario touched. Our Legacy Radar turned “are we done with the old system here?” into something you can see.
Internal tools rarely make it into portfolios or conference talks, but this one moved a 50-module migration more than most features did. If you’re staring down a long strangler migration, build your scenario runner first — it’s the instrument panel for everything that comes after.
Questions about migrations or internal tooling? Reach out.