Replacing a Critical Data Path Without a Flag Day

Replacing an API call is easy. Replacing the source of truth behind an automated decision is not.

I was reminded of this while migrating a critical workflow from a legacy event feed to a canonical state API. Both systems appeared to answer the same question: should the workflow act on this record now? But they had different schemas, different update timings, and, more importantly, slightly different models of the same lifecycle.

This was not a path where we could deploy the new code on Friday and watch the error rate. A false negative could leave work undone. A false positive could trigger an irreversible action from stale data. The HTTP request succeeding told us almost nothing about whether the new path was making the right decision.

So we did not treat it as a normal code replacement. We treated it as a controlled transfer of authority.

The migration used a query-only mode, shadow comparisons, discrepancy alerts, and a deliberately boring cutover. None of those techniques are particularly novel. What mattered was how we combined them, what we chose to compare, and how we decided the old path was finally safe to delete.

This post walks through that process.

The dangerous assumption: same data, newer API

The legacy path consumed a source built specifically around one class of lifecycle event. The replacement exposed a broader canonical record containing current state, relevant dates, and other attributes used by several workflows.

On paper, the migration looked like this:

That diagram hides the risky part. The two sources did not merely encode the same fact using different field names.

A purpose-built event feed tends to answer an event-shaped question: which transitions were recorded? A canonical API tends to answer a state-shaped question: what is true about this entity now?

Those questions overlap, but they are not identical.

An entity can have an old end-state event and later return to an active state. A record can contain several dates, each valid in its own business context. One source may update immediately while another catches up later. Missing data can mean “not applicable,” “not received yet,” or “something is broken.”

If we had translated fields one-for-one and switched traffic, the code would have looked correct while preserving none of those semantics.

The first useful decision was therefore to stop calling this an API migration. We were migrating a business decision from one model of the world to another.

Start by writing down the invariants

Before building the new path, we wrote down what the workflow must continue to guarantee.

The important invariants were roughly:

• An active entity must never be processed because of a stale historical event.
• A legitimate lifecycle transition must not be missed because one optional field is absent.
• The effective date must have the same business meaning before and after migration.
• Ambiguous or contradictory data must fail safely and become visible.
• Reprocessing the same record must not produce duplicate downstream actions.

This step sounds obvious, but it changed the review entirely. Instead of asking whether the new client correctly parsed a payload, we could ask whether it preserved the rules the system existed to enforce.

It also exposed a subtle problem with “the old system is the source of truth.” If the legacy path had known defects, perfect agreement would reproduce them. The old output was a baseline, not an oracle.

That meant every mismatch needed investigation, but it did not mean the new path was automatically wrong.

Phase one: make the new path incapable of causing damage

The first version of the new integration was deliberately incomplete. It could query the canonical state API, normalize the response, and calculate what action it would take. It could not perform that action.

I think of this as query-only mode:

decision = evaluate_canonical_record(record)

if query_only:
    return decision

execute(decision)

The real implementation had more safeguards than this, but the boundary was just as explicit. Read and decide on one side; mutate on the other.

Query-only mode gave us a few useful properties:

• We could run the code with realistic production data.
• We could inspect decisions without creating tickets, sending notifications, or changing access.
• We could debug authentication, pagination, missing fields, and schema assumptions separately from cutover risk.
• We had a reusable operational tool for investigating individual records.

This last point was unexpectedly valuable. Migration controls are often treated as temporary scaffolding, but a safe read-only execution mode is also a good diagnostic interface. It lets an engineer ask, “What would the system do with this input right now?” without having to reproduce the entire workflow locally.

There was one rule we kept firm: query-only could not mean “mostly read-only.” If a code path still emitted an event or called a downstream service before checking the flag, the control was cosmetic. The no-side-effect guarantee had to sit at the boundary where side effects began.

Phase two: one writer, two decision-makers

Once the new path could evaluate real records safely, we ran it alongside the legacy implementation.

Only the legacy path was allowed to perform actions. The new path observed the same logical input and produced a candidate decision. We then compared the two.

This is usually called a shadow migration or dark launch. The new code sees production-shaped traffic but does not own production effects.

The obvious implementation compares two Boolean values:

legacy_should_process == new_should_process

That is useful, but not enough. When the values differ, a Boolean tells you that the migration is unsafe and nothing about why.

We made the decision explain itself:

Decision(
    should_process=False,
    effective_date=None,
    reason="currently_active",
    source_record_id="...",
)

Now the shadow result could compare:

• Whether each path would act.
• Which effective date it would use.
• Why it reached that conclusion.
• Which source record contributed to the decision.
• Whether either path considered the input incomplete or ambiguous.

This is the part of the migration I would reuse everywhere: compare normalized decisions, not raw responses.

Raw payload comparison is noisy. One API may use null where another omits a field. Dates may use different time zones. Identifiers may refer to different resources. A hundred harmless representation differences can hide the one semantic difference that matters.

The normalized decision is the contract your users actually experience.

A mismatch is a finding, not just an error

As soon as the comparison ran against real data, discrepancies appeared. That was the point.

It is tempting to turn every mismatch into a page. I would avoid that. Early shadow traffic can be noisy, and training people to ignore an alert stream is a poor way to launch a critical system.

Instead, we recorded every mismatch with enough context to investigate it and grouped them into a small taxonomy:

1. Expected timing differences - one source had updated before the other.
2. Representation differences - the sources agreed, but normalization was incomplete.
3. Semantic differences - both records were valid, but the business interpretation differed.
4. Data-quality problems - records were missing, stale, or internally contradictory.
5. Implementation defects - our new code had selected the wrong field or applied the rule incorrectly.

This classification matters because each category leads to a different response.

A timing difference may need a grace period or a later recheck. A representation problem belongs in the adapter. A semantic difference needs a product or domain decision. Bad source data needs a defensive rule and an escalation path. An implementation bug needs code and a regression test.

Without a taxonomy, teams tend to “fix the diff” until the graphs become green. That can accidentally teach the new system to mimic legacy behaviour without understanding it.

The edge cases were the migration

The happy path agreed quickly. That did not make the migration nearly complete.

Two edge cases forced us to refine the new model.

The first involved a historical end-state event for an entity whose current state had since changed. If we looked only for the existence of that event, we could produce a false positive. The canonical record gave us another signal: the entity’s current status. We added that verification before allowing the workflow to proceed.

The second involved choosing the effective date. The new source exposed more than one plausible date, and the most conveniently named field was not necessarily the date the downstream process expected. We had to trace the business meaning through the old path and deliberately select the corresponding value.

Neither bug was difficult to fix once understood. The hard part was creating a migration that allowed us to see them before they became actions.

That is why I do not judge shadow migrations by the amount of traffic replayed. A million ordinary records can give more confidence than they deserve. One reactivation, one delayed update, or one contradictory date can tell you much more about whether the new model is correct.

For this kind of workflow I want an explicit scenario set:

• Normal lifecycle transition.
• Future-dated transition.
• Reactivation or status reversal.
• Missing optional attributes.
• Conflicting dates.
• Duplicate input.
• Source timeout or partial response.
• A record that changes while being processed.

Some scenarios will occur naturally during the shadow period. Rare but dangerous ones should be exercised with fixtures or controlled replay rather than waiting for production to provide them.

Deciding when to cut over

“The dashboards look fine” is not a cutover criterion.

Before transferring authority to the new path, we wanted evidence in several dimensions:

• Correctness: no unexplained decision or effective-date mismatches.
• Scenario coverage: important lifecycle transitions had been observed or tested.
• Stability: the comparison stayed clean across a meaningful observation window, not just one quiet day.
• Failure behaviour: timeouts, missing records, and contradictory data failed safely.
• Observability: we could tell whether the new path queried, decided, skipped, failed, or acted.
• Rollback: restoring the old authority was understood and quick.

There is no universal percentage or number of days that makes a migration safe. The right window depends on the frequency and consequence of the events you are trying to observe.

If an important edge case happens once a month, a clean afternoon tells you nothing about it. If the cost of a false positive is high, the threshold should reflect that asymmetry.

The cutover itself was intentionally uneventful. We changed which path was authoritative while retaining the ability to compare and roll back. We did not bundle unrelated cleanup into the same release. Boring is a feature when transferring control of a critical workflow.

Do not leave the old path “just in case”

After the new path had operated successfully, we removed the legacy implementation and the shadow comparison.

This can feel premature. Keeping the old path around appears to preserve a fallback. In reality, a dormant fallback decays quickly:

• Its credentials and dependencies still need maintenance.
• Engineers must continue reasoning about two implementations.
• Future changes may update one path but not the other.
• Someone can accidentally reactivate code that has not been exercised in months.
• The temporary feature flag becomes permanent architecture.

A rollback path is valuable during migration. A second production system with no clear retirement date is technical debt.

We treated deletion as a planned migration phase rather than optional cleanup:

The order matters. Deleting before the soak period removes your quickest recovery option. Never deleting leaves you paying for the migration forever.

The comparison infrastructure should also be removed or deliberately repurposed. Shadow code often doubles reads, emits high-cardinality logs, and contains branching that the main workflow no longer needs. Once its question has been answered, it should not quietly become part of the permanent request path.

What I would carry into the next migration

The practical pattern is straightforward:

1. Define the invariants before translating fields.
2. Separate reads and decisions from side effects.
3. Give the new path a genuine query-only mode.
4. Keep exactly one writer while both paths evaluate.
5. Compare normalized decisions and their reasons.
6. Investigate and classify every meaningful mismatch.
7. Measure coverage across business scenarios, not only traffic volume.
8. Set evidence-based cutover and rollback criteria.
9. Keep the cutover small.
10. Delete the legacy path after a defined soak period.

The broader lesson is that migrations like this are not primarily data plumbing exercises. They are exercises in transferring trust.

The old path has years of accumulated behaviour, including behaviour nobody documented because the code made it seem obvious. The new source may be cleaner and more canonical, but that does not make your interpretation of it correct. Confidence comes from forcing both systems to make their decisions in the open, then explaining every place they disagree.

That takes longer than changing an endpoint. It is still much cheaper than discovering after cutover that a green deployment was making the wrong decision perfectly.

Thanks for reading ✌️