The failover agent that mistook no answer for no action

Northstar Ledger (a fictional B2B commerce-infrastructure company) routes checkout and inventory updates for regional grocery chains. Its customers do not run one Northstar checkout page. They use Northstar behind their own mobile apps, self-checkout kiosks, and delivery sites.

At the Saturday peak, those systems send roughly 48,000 order operations per minute. A short outage creates queues. An incorrect recovery can create something worse: orders accepted through a path that the company no longer believes is authoritative.

Northstar stored checkout state in an Amazon Aurora PostgreSQL global database. The primary Region served writes. A secondary Region stayed ready for disaster recovery. The operating rule was strict: Northstar could advertise only the writer confirmed by Aurora's current topology.

Imani (the Staff Reliability Engineer responsible for Northstar's database-recovery control plane) had spent the previous six months reducing the time between an incident page and a safe recovery decision.

Her team built Relay, a bounded AI incident worker. Relay read approved telemetry, compared an incident with reviewed runbooks, and proposed a typed recovery plan. It could not promote a database or change traffic. A deterministic policy service checked every proposal. A human incident commander then approved the exact target and the maximum tolerated data loss before any effect was allowed.

This mattered because Relay was useful precisely where incidents were messy. It could gather replication lag, recent deployments, health probes, and runbook constraints in seconds. It could explain why one recovery target fitted the evidence better than another.

It could not make an ambiguous cloud operation unambiguous by thinking harder.

How a checkout normally reaches the writer​

Before looking at the failover, it helps to place Relay beside the customer path. A shopper's request normally passes through Northstar's regional endpoint and order services before Aurora accepts the write. Relay is not in that request path. During an incident, it feeds a separate recovery control path that can change the Aurora writer and the Route 53 target serving future requests.

The recovery path they trusted​

The system around Relay looked cautious.

The incident console was React. A NestJS service called incident-api assembled the request. A Python relay-worker called the language model and returned a typed proposal, not free-form shell commands.

A Temporal workflow on Amazon EKS owned the recovery sequence. Temporal activity workers sent heartbeats while they ran. Kubernetes Leases helped the team identify the current controller. PostgreSQL stored the incident, plan, policy result, and approval. A separate NestJS service called effect-gateway owned the AWS credentials and exposed narrow RDS and Route 53 operations.

No Relay worker held a general AWS credential. AWS STS minted short-lived credentials for the gateway's adapters. OpenTelemetry joined traces across the workflow, and CloudTrail recorded AWS API activity.

That was a competent design. The team had durable state, bounded tools, least-privilege credentials, deterministic policy, and a human approval gate.

The recovery activity still contained one quiet mistake. The naive workflow placed the promotion and its follow-on Route 53 update inside one coarse, heartbeating recoverRegion activity. That allowed overlapping activity attempts to perform both effects. If traffic had been a later workflow-scheduled activity, worker A could not have advanced to it independently after the first activity timed out.

When Temporal retried an activity, that attempt generated a fresh request key. The gateway checked whether the caller was permitted to invoke the adapter, but it did not keep one authoritative record for the logical promotion. It also did not reject an old worker by workflow epoch.

The team's mental model was:

Temporal remembers the step. Kubernetes knows which worker is current. If a call fails, retrying the activity will either recover the result or safely perform the same work.

They had discussed duplicate requests. They had not designed for a request that succeeded elsewhere while leaving no answer locally.

Saturday at 02:13​

At 02:12, the primary Region developed a network impairment. Checkout latency climbed. Health probes disagreed by source Region. Replication lag to the secondary remained inside Northstar's approved emergency loss bound, but the primary was becoming unreachable from enough application nodes to threaten the order queue.

At 02:13:00, Relay assembled the evidence. Its proposal named the secondary cluster, measured lag, included Northstar's digest and observation time for the Aurora topology snapshot, and stated the maximum loss bound. It recommended the reviewed regional-promotion runbook.

The proposal was correct.

At 02:13:18, Darius (the incident commander with authority to approve database recovery) reviewed the evidence and approved that exact digest. He did not approve “fix checkout.” He approved one target under one stated loss envelope.

The policy service accepted it. The Temporal workflow recorded it. Activity worker A began the promotion.

At 02:13:24, worker A's recoverRegion handler created local request key attempt-a and called the RDS adapter through effect-gateway. The gateway sent the approved promotion request to AWS.

At 02:13:27, AWS accepted the promotion and returned an acknowledgement to worker A's handler. Before the handler could persist that receipt or report its activity result to Temporal, its connection to Northstar's control services failed. The acknowledgement survived only in the still-running process's memory.

The workflow history now contained an honest fact: no result had been recorded.

It did not contain another honest fact: the promotion had already crossed Northstar's boundary and AWS had acknowledged it to the isolated handler.

At 02:13:33, Temporal's configured activity heartbeat timeout expired. Under the activity's retry policy, Temporal scheduled another attempt, which worker B polled. Independently, Northstar's controller Lease expired and B became the current controller. Neither mechanism terminated worker A's local activity handler.

At 02:13:40, worker B loaded the durable workflow history, saw the missing activity result, and created request key attempt-b. Its first Northstar-cached topology observation was stale: it still named the impaired original cluster as writer. Following the recovery code, B retried the coarse activity and prepared a “hold on the known writer” traffic fallback toward the original cluster.

The replacement worker was not ignoring the checkpoint. It was following it.

At 02:13:46, Aurora's authoritative topology identified the secondary cluster as the writer. The original promotion had happened. The missing response had concealed an accepted effect, not a failed one.

At 02:13:50, worker A's still-running recoverRegion handler regained control-service connectivity and continued to its embedded Route 53 call. It submitted change A-184, pointing checkout to the newly promoted secondary cluster. Temporal could reject A's late activity completion; it could not undo an external call the handler had already made. Nothing at effect-gateway rejected A's next action as stale.

Worker A and worker B now held different beliefs about the same recovery:

• worker A held the unpersisted AWS acknowledgement and intended to point traffic to the newly promoted secondary cluster;
• worker B held a stale snapshot naming the impaired original cluster as writer and intended to point traffic back there.

Both could still begin a new Route 53 action.

At 02:14:22, after change A-184 had left provider processing, Route 53 accepted B's sequential change B-771, pointing the same record back to the original cluster. At 02:21:08, after joining Aurora's writer topology with both change records, the incident commander approved repair change R-009 toward the promoted secondary. Route 53 processed the three batches in submission order; Northstar's contradictory controllers, not Route 53, had created the oscillation.

Long-lived application connections did not all move when DNS changed. Some clients held existing pools while others resolved the newer target. Checkout moved to the promoted secondary, back toward the impaired original cluster, then finally to the authoritative writer.

Aurora did not casually create two managed-database primaries. Its topology converged on one writer. The incident was in Northstar's recovery control plane: two workers were allowed to issue traffic-changing effects from different beliefs about that topology.

The checkout target changed Regions three times. The order service placed 11,842 operations into its retry queue. A total of 1,906 shoppers saw a temporary checkout warning.

At 02:23, the workflow reached a green terminal state. Aurora was healthy. The final traffic target matched the final writer. The top-level recovery dashboard called the run successful.

The customer path had still been unsafe.

Three histories, all telling the truth​

Imani did not start with the model transcript. She put three timelines side by side.

The first was Relay's proposal history. It showed the evidence Relay had used, the chosen secondary, Northstar's observed-topology digest and timestamp, the loss bound, and Darius's approval digest. Nothing in it explained the oscillation. Relay had recommended the right action.

The second was Temporal's durable workflow history. It showed the coarse recoverRegion activity scheduled, worker A's heartbeat loss, no recorded receipt, a timeout, and a replacement attempt. That record was also correct.

This was state persistence and checkpointing+: the workflow had preserved its own progress well enough to resume. A checkpoint could say what had been scheduled, observed, or recorded. It could not prove what AWS had done after the request crossed the network.

The third was external authority. Aurora's current topology established which cluster was the writer. CloudTrail later corroborated that the original API call reached AWS. Route 53 held three sequential change resources, each with its own status and timestamps. Those records showed effects the workflow had not understood when it retried.

The contradiction disappeared once the team stopped asking which history was wrong.

They were histories of different things.

The phrase from the design review—“durable means exactly once”—finally sounded as dangerous as it was.

Exactly-once semantics+ can exist inside a stated boundary when the participants cooperate through atomic state changes, deduplication, or idempotency. Temporal could durably record and replay workflow history. PostgreSQL could make one ledger transition atomic. Neither mechanism could silently include an arbitrary AWS operation that had already crossed the network.

The timeout meant only this: Northstar did not know the result yet.

The key had named the attempt​

The code review made the first bug painfully small:

// Tempting: every retried activity invents a new intent.
async function promoteActivity(plan: ApprovedPlan) {
  return effectGateway.promote({
    ...plan,
    requestKey: crypto.randomUUID(),
  });
}

// Correct boundary: allocate this record once, before dispatch.
async function resumePromotion(operationId: string, epoch: number) {
  const op = await operations.get(operationId);

  switch (op.status) {
    case 'CONFIRMED':
      return op.receipt;
    case 'DISPATCHING':
    case 'UNKNOWN':
      return reconcileWithAws(op);
    case 'AUTHORIZED':
      return effectGateway.dispatch({
        operationId: op.id,
        parameters: op.parameters,
        approvalDigest: op.approvalDigest,
        topologySnapshotDigest: op.topologySnapshotDigest,
        topologyObservedAt: op.topologyObservedAt,
        fencingEpoch: epoch,
      });
    default:
      return stopForReview(op);
  }
}

An idempotency key+ should identify one logical operation, not one transport attempt. Reusing a stable identity gives a cooperating receiver the chance to recognize a repeat within its documented scope and retention window. Generating attempt-b told the gateway that worker B had brought new intent.

But replacing randomUUID() with a stable value was not the whole fix.

Not every provider operation exposes the same idempotency or lookup contract. The RDS promotion path still needed reconciliation against Aurora's current topology and AWS evidence. Route 53 returned a change ID that Northstar could persist and poll from PENDING to INSYNC. Each adapter needed to respect the destination's actual contract.

An idempotency key could help suppress a duplicate dispatch. It could not tell Northstar whether an unobserved operation had completed, whether the world had changed since approval, or whether an old worker was still alive.

An operation could be unknown​

Imani's team added a PostgreSQL operation ledger in a failure-independent Region. Before any effect, the workflow created one stable operation_id and bound it to:

• the exact effect type and target;
• a canonical digest of its parameters;
• the policy version and approval digest;
• Northstar's digest and observation time for the Aurora topology snapshot;
• the current workflow epoch;
• any AWS request, audit, resource, or change identifiers returned later.

The ledger did not force every operation into SUCCEEDED or FAILED. A normal dispatch could confirm directly; an ambiguous one branched into an explicit uncertainty state:

PROPOSED → AUTHORIZED → DISPATCHING → CONFIRMED
                                  ↘ UNKNOWN
                                      ├→ CONFIRMED
                                      ├→ SUPERSEDED
                                      └→ MANUAL_REVIEW

It also allowed REJECTED before dispatch. SUPERSEDED and MANUAL_REVIEW were resolved outcomes for an uncertain operation, not mandatory stops on the successful path.

UNKNOWN was not an error message. It was a statement about evidence. The call might have failed before leaving Northstar. It might have been accepted and still be running. It might have completed while its response disappeared. Until reconciliation established more, all three remained possible.

For an unknown promotion, the replacement worker did not manufacture a new logical operation. It read Aurora's current global-cluster topology, joined CloudTrail and request evidence where available, checked the intended target, and recorded the observation time. Only then could it confirm the original operation, retry under the same safe identity when the contract allowed it, supersede it, or stop for manual review.

The next action faced a second barrier. Before changing traffic, the workflow re-read topology, policy, and approval. Northstar's topology digest supported this changed-world check; it was not an RDS ETag, resource version, or atomic API precondition. The checkpointed plan remained valuable historical evidence. It was not present authority.

The team gave each Route 53 change a separate operation identity. The gateway stored the returned change ID and polled GetChange. PENDING meant AWS had accepted a change that had not yet propagated. INSYNC established propagation through Route 53's authoritative DNS servers. Northstar still required regional DNS-resolution checks and connection-pool turnover before declaring the customer path complete.

A lease was not a fence​

The Kubernetes Lease had done its job. It helped the cluster choose worker B after worker A stopped sending heartbeats.

It had not revoked worker A's memory, stopped its process, cancelled an in-flight AWS request, or told Route 53 to reject its next call.

Northstar added a monotonically increasing fencing token+—called a fencing epoch in the implementation—to every claimed dispatch. The gateway atomically claimed the tuple {operation_id, epoch, effect} in the ledger. After a successful claim, the gateway—not the worker—owned that dispatch. A newer epoch prevented another claim but could not cancel a call already claimed or sent; an ownership change during dispatch moved the operation to UNKNOWN for reconciliation.

When worker B took epoch 240, a later request from worker A carrying epoch 239 was rejected at the gateway.

That boundary was deliberately narrow:

• it stopped a stale worker from initiating a new effect through Northstar's gateway;
• it did not cancel a request already sent to AWS;
• it did not turn Aurora or Route 53 into participants in Northstar's transaction;
• it did not remove the need for provider request IDs, topology checks, or reconciliation.

Workers still received no general RDS or Route 53 credentials. If a worker could bypass the gateway, the fence would be a number in a log rather than an enforced safety mechanism.

The system they took back to review​

The corrected design separated four ownership boundaries.

The human and incident interface contained the React console, incident-api, Relay's evidence, and the incident commander's exact approval.

The AI analysis and durable workflow boundary contained the Python relay-worker, Temporal, Kubernetes controller election, and versioned activity workers on EKS. Relay proposed. Temporal remembered orchestration. The Lease helped choose a current controller. None of them owned AWS truth or enforced the effect fence.

The operation control and evidence boundary contained the failure-independent PostgreSQL ledger, deterministic policy service, effect-gateway, reconciliation workers, OpenTelemetry, and CloudTrail correlation. This was where stable operation identity, current epoch, approval binding, and effect evidence met.

The AWS data and traffic boundary contained Aurora's authoritative topology and Route 53's asynchronous change resources. Northstar observed those systems through their own contracts rather than treating its last checkpoint as a copy of them.

The recovery rule became simple enough to print beside the incident console:

Resume the task from durable history. Reconcile an uncertain effect with its authority. Revalidate the world before beginning the next effect.

Simple did not mean cheap.

The tests that cared how it finished​

The original dashboard evaluated the end state: was Aurora healthy, was one writer present, did the workflow finish, and did checkout recover?

Those questions still mattered. They were not enough.

Imani added Trajectory vs end-state evaluation+. End-state evaluation checked where the system finished. Trajectory evaluation checked the observations, approvals, dispatches, policy decisions, and prohibited intermediate transitions used to get there.

The test environment repeatedly injected the failures that ordinary happy-path tests hid:

• drop the provider response after dispatch;
• terminate a worker before it persists an external receipt;
• stop heartbeats, start a replacement, then wake the old worker;
• duplicate or delay audit events;
• change topology between approval and dispatch;
• deploy a new worker build while an old execution remains open;
• leave a Route 53 change PENDING longer than expected;
• let an operator modify the same recovery resource.

Deterministic graders checked one stable promotion intent, no new stale-epoch dispatch, and no completion without authoritative evidence. Every UNKNOWN operation had to become reconciled or explicitly escalated. The final writer and traffic target had to agree. A run failed if traffic made a prohibited transition, even when the final dashboard was green.

Model evaluation ran separately. Relay was graded on evidence classification, loss-bound extraction, and plan proposal. Improving those scores could not compensate for an unsafe effect trajectory.

Across Northstar's 1,200 fictional fault-injection trials, every unknown operation either reconciled to authoritative evidence or stopped in MANUAL_REVIEW; zero stale-epoch dispatches crossed the gateway. None were converted into a fresh logical promotion merely because a response was missing.

The result that mattered most was also the least attractive on a status slide: safe write resumption became slower.

What the safety cost​

The previous path optimized for recovery time. If the first call truly had failed before reaching AWS, an immediate replay was fast.

The new path deliberately paused to establish external truth. During ambiguous cases, Northstar kept checkout read-only for another 45 to 90 seconds. More orders entered a queue instead of receiving an immediate answer.

The team now operated a failure-independent control database, an effect gateway, provider-specific reconcilers, epoch management, and richer evidence retention. A damaged control store could delay recovery. A conservative precondition could send an incident to a human even when an automatic action would have worked.

Operations accepted that cost because the alternative spent correctness without measuring it. Product accepted it because a visible “please retry” message was recoverable. Conflicting order histories were not.

The design did not promise exactly-once cloud effects. It promised something more testable:

• one stable intent;
• explicit uncertainty;
• no new action from a stale owner;
• no completion claim without evidence from the system that owned the result.

The question Imani used next​

The incident changed how Imani reviewed every durable agent.

For a payment gateway, she asked what happened when capture succeeded but its response disappeared.

For a secret-rotation worker, she asked which credential actually existed when a checkpoint and the target account disagreed.

For a deployment agent, she asked whether the approved plan still described the cluster after a human changed it during a pause.

The surface changed. The review question did not:

If this worker wakes with no answer, what proves whether the outside world already acted?

A durable workflow can remember where its computation stopped. It cannot remember a fact it never received.

That fact has to be recovered from the authority that owns it.

The production topology​

The incident sequence earlier showed when Northstar's histories diverged. The final deployment view shows where each responsibility lives, and lets the reader switch between the failed path and the corrected control boundary.

Northstar Ledger, its people, its incident, and its metrics are fictional. Technical grounding: AWS guidance on idempotency and retries in durable execution, Kubernetes Leases, the Amazon RDS global-cluster failover API, Route 53 change batches, Route 53 change status, and resource freshness validation. These sources establish technical behavior and boundaries; they do not describe the fictional incident.