The retry that made the outage worse
Kiteframe Pay (a fictional payment-risk infrastructure company) helps online marketplaces decide whether a card order should be approved, declined, or reviewed by a human. Merchants call Kiteframe during checkout, before they capture money or promise inventory to a shopper.
Most orders never need generative AI. Deterministic rules settle obvious cases in under 80 milliseconds. The difficult eight percent—new devices, unusual delivery patterns, sparse account histories—enter Aster, Kiteframe's AI-assisted risk analyst.
Aster was not a chatbot and could not approve a payment. It gathered approved evidence, used a language model to produce a typed risk recommendation with cited signals, and passed that recommendation to a deterministic merchant-policy engine. Without Aster, those ambiguous orders went to a manual-review queue. During a large sale, that queue could grow faster than Kiteframe's analysts could empty it.
At 02:13 on Tuesday, model latency rose sharply. Seven minutes later, Kiteframe's entire checkout success rate had fallen from 99.4% to 71%.
The model provider was recovering.
Kiteframe's retries would not let it.
Where AI sits in a checkout
Mina Rao (the Principal Reliability Engineer responsible for Kiteframe's risk-decision platform) began the incident by drawing the useful path, not the failure. One merchant request entered risk-api. A bounded Python worker assembled evidence. The model returned a schema-constrained recommendation. A policy service—not the model—owned the payment decision.
The architecture below lets you switch between that normal journey, the amplification incident, and the protected design. The numbered explanation remains readable on a phone; full-screen mode exposes the production boundaries and zoom controls.
Concepts in this story8 concepts
A limit on how long a caller will wait before treating an operation as failed. A timeout ends the wait; it only stops the underlying work when cancellation is propagated.
A new attempt after a failed one. Retries help with transient faults, but they add load and can repeat side effects unless the operation is safe to retry.
A retry schedule in which the delay grows exponentially after each failure, usually up to a cap. It reduces pressure on a dependency that is still unhealthy.
Random variation added to retry delays so many clients do not retry in lockstep. It spreads recovery traffic over time instead of creating another spike.
A feedback loop where failed requests trigger enough retries to raise load, deepen the failure, and delay recovery.
A stateful guard that stops calls to a failing dependency for a limited time, then probes for recovery. It protects both systems; it is not another retry mechanism.
Closed allows calls and records outcomes. Open rejects calls immediately. After a cooldown, half-open admits a small number of probes and uses their results to close or reopen the circuit.
A deliberately reduced result returned when the preferred dependency or path is unavailable—for example, cached data without personalization.
Three teams had each added a sensible retry
Kiteframe's merchant-facing services ran on Amazon EKS. A NestJS checkout-api called risk-api; a Python risk-agent assembled device, account, and merchant evidence; and an internal AI gateway called the external model API. Envoy sidecars handled service-to-service networking. PostgreSQL held merchant policies. Amazon SQS held manual-review cases. OpenTelemetry joined an order ID to traces across the path.
Each component had been tested in isolation.
Three retry policies had accumulated over eighteen months:
checkout-apiretried the whole risk evaluation once if it had no answer after 2.4 seconds.risk-agentallowed three generation attempts because transient model failures were common during deploys.- Envoy retried selected connection resets and HTTP 503 responses once.
No team had intended twelve model calls. Yet the theoretical multiplication for one risk decision was:
2 whole evaluations × 3 agent attempts × 2 transport attempts = 12 provider requests
A is another attempt after a failure. The word “another” sounds harmless until several layers are each allowed to manufacture attempts for the same original work.
The policies also disagreed about time.
The merchant request had a four-second deadline. checkout-api spent 2.4 seconds before its retry. Each agent attempt allowed two seconds. Envoy knew nothing about either budget. A later layer could therefore begin work that had no chance of returning before the merchant had left.
At the model boundary, Kiteframe's 2-second stopped its local wait. It did not prove the provider had stopped computing. Cancellation was not propagated through every hop, and some requests had already entered the provider's queue. Their tokens continued to consume capacity even though Kiteframe would discard the late answers.
That distinction changed the arithmetic. The dashboards counted timed-out calls as finished client work. The provider still counted them as admitted inference.
The autoscaler made the breaker forget
At 02:13, the provider's 99th-percentile latency rose from 1.1 seconds to 5.8 seconds and some requests returned 503. Kiteframe was receiving roughly 390 AI-routed orders per second.
By 02:16, it was starting more than 3,400 provider requests per second. The exact amplification varied because some calls completed and not every response qualified for every retry. The direction did not vary:
- queues increased latency;
- latency crossed more local timeouts;
- timeouts created more attempts;
- those attempts deepened the queues.
The recovery mechanism had become a .
The team believed it already had a . After twelve failures, each risk-agent pod opened its local breaker and stopped calling the model for thirty seconds.
Then Kubernetes did exactly what Kiteframe had configured it to do. CPU and in-flight work rose, so the Horizontal Pod Autoscaler expanded the deployment from 24 pods to 78. Each new pod started with an empty memory and a closed breaker. It admitted model traffic while older pods were rejecting it.
The breaker had protected one process. The dependency was being attacked by a fleet.
At 02:18, the provider restored enough capacity for about 2,200 request starts per second. Kiteframe was still offering far more. A health chart turned green for forty seconds, then red again as delayed retries and new pods arrived together.
Mina stopped the autoscaler at 78 replicas. That prevented another wave of fresh breaker state, but it did not establish which layer owned retries or how much work the fleet was permitted to admit.
The trace looked like a tree, not a request
The incident review began with one order, ord_81K2.
Its root trace had two risk.evaluate spans. Beneath them were six generation spans. Transport telemetry showed eleven provider request starts; one connection had failed before Envoy could send a body. Four model requests finished after their parent spans had timed out. Two produced identical valid recommendations that nobody read.
Nothing in Aster's reasoning caused the outage. The prompts were valid. The tool evidence was current. The typed responses passed schema validation.
AI was still load-bearing in the incident because model inference was the scarce, slow, externally scheduled work being amplified. Replacing the model with a database call would have changed the cost and latency profile. It would not have fixed retry ownership.
Mina's first rule for the repair was therefore architectural:
One customer intent gets one deadline and one retry budget. Crossing a boundary does not create more of either.
One place was allowed to try again
Kiteframe moved model-attempt policy into the AI gateway. The other layers stopped retrying generation.
risk-api created an absolute deadline and an attempt budget when it accepted the order. risk-agent inherited them. Envoy retries were disabled for the model cluster. The AI gateway could make one additional attempt only for a narrow set of transient failures, only when budget remained, and only when enough deadline remained for the result to be useful.
The delay used capped . Its maximum grew after a failure. The actual wait used full , choosing a random delay below that cap so every pod did not wake on the same boundary.
async function generateRiskRecommendation(ctx: DecisionContext) {
// Allocated once at ingress; inherited by every downstream hop.
const budget = ctx.retryBudget;
const deadline = ctx.deadline;
while (budget.tryConsumeAttempt()) {
const remainingMs = deadline.remainingMs();
if (remainingMs < MIN_USEFUL_ATTEMPT_MS) break;
const permit = await aiAdmission.tryAcquire({
merchantId: ctx.merchantId,
deadline,
});
if (!permit) break;
try {
return await model.generate(ctx.prompt, {
signal: deadline.abortSignal(),
timeoutMs: Math.min(1_600, remainingMs),
});
} catch (error) {
if (!isTransient(error) || !budget.canRetry()) throw error;
const capMs = Math.min(200 * 2 ** budget.attemptsUsed, 1_200);
await sleep(Math.random() * capMs, deadline.abortSignal());
} finally {
permit.release();
}
}
return degradedDecision(ctx);
}
The abort signal made cancellation explicit through Kiteframe's own code. It still could not guarantee that a remote provider would erase already-admitted work. The gateway therefore treated cancellation as a capacity hint, not as proof of zero downstream cost.
A breaker for the destination, not the pod
The repaired gateway maintained one logical model-destination policy across its replicas. A Redis-backed token bucket enforced a fleet-wide concurrency ceiling and fair per-merchant admission. If Redis was unavailable, the AI path failed closed into degraded mode instead of silently admitting unbounded inference.
The breaker's now described the gateway's relationship with the model destination:
- Closed: admit calls up to the concurrency limit and measure outcomes.
- Open: admit no normal calls; return degraded decisions immediately.
- Half-open: after a cooldown, allow at most twenty probes per second across the fleet. Successful probes closed the circuit gradually; failures reopened it.
This design introduced coordination cost. Redis became part of the admission path. Shared state could be delayed. A breaker threshold could open during a brief provider wobble and reject work that might have succeeded.
Kiteframe accepted those costs because the alternative allowed every new pod to rediscover the same outage by spending provider capacity.
The fallback needed its own capacity model
Opening the breaker did not make risk disappear.
For established merchants and low-risk returning shoppers, the policy engine could use a stricter deterministic rule set. New or conflicting identities went to SQS for human review. Checkout displayed a pending-review state instead of pretending an AI recommendation existed.
That was a : a deliberately reduced service when the preferred path was unavailable.
The first version was operationally unsafe. It could enqueue 390 cases per second while the review team could resolve only 42. A “successful” fallback would merely move the outage to tomorrow's queue.
Product, fraud operations, and reliability set an explicit degraded-mode budget:
- deterministic policy could approve only a reviewed low-risk segment;
- the manual queue admitted cases until its projected age reached forty minutes;
- after that bound, remaining ambiguous checkouts received an honest retry-later response;
- large merchants received fair shares so one flash sale could not consume the whole review team.
During the next provider impairment, 404 ambiguous orders per second reached Kiteframe. The gateway capped provider starts at 520 per second. When the breaker opened, only twenty global probes per second tested recovery. Checkout remained available for deterministic decisions, while 31,406 orders entered review and 6,218 received retry-later responses.
Conversion for the affected segment fell 4.8%. Fraud operations worked an elevated queue for three hours.
Those were not footnotes to the technical solution. They were the price of keeping the system inside known capacity while its preferred intelligence was unavailable.
The production path after the incident
Switch the architecture to Protected path to see the final ownership boundaries. The most important component is not the retry loop. It is the single admission point before expensive inference, backed by a deadline that still represents the merchant's request.
What Mina carried into the next review
A timeout says, “this caller stopped waiting.” It does not say, “the work stopped.”
A retry can improve availability when the failure is transient, the operation remains useful, and one layer owns the policy. Repeated independently across a call graph, it can turn a partial provider impairment into a platform outage.
A circuit breaker protects the boundary where it observes and rejects work. If every autoscaled process owns a fresh breaker, the fleet may have no breaker at all.
And a fallback is not free availability. It is another production system with a quality limit, a queue, and a finite team behind it.
Concepts in this story8 concepts
A limit on how long a caller will wait before treating an operation as failed. A timeout ends the wait; it only stops the underlying work when cancellation is propagated.
A new attempt after a failed one. Retries help with transient faults, but they add load and can repeat side effects unless the operation is safe to retry.
A retry schedule in which the delay grows exponentially after each failure, usually up to a cap. It reduces pressure on a dependency that is still unhealthy.
Random variation added to retry delays so many clients do not retry in lockstep. It spreads recovery traffic over time instead of creating another spike.
A feedback loop where failed requests trigger enough retries to raise load, deepen the failure, and delay recovery.
A stateful guard that stops calls to a failing dependency for a limited time, then probes for recovery. It protects both systems; it is not another retry mechanism.
Closed allows calls and records outcomes. Open rejects calls immediately. After a cooldown, half-open admits a small number of probes and uses their results to close or reopen the circuit.
A deliberately reduced result returned when the preferred dependency or path is unavailable—for example, cached data without personalization.
Kiteframe Pay, its people, and its metrics are fictional. Technical grounding: AWS Builders' Library on timeouts, retries, backoff, and jitter, Envoy retry and circuit-breaking configuration, Kubernetes horizontal pod autoscaling, and OpenTelemetry context propagation.