Skip to main content

One post tagged with "Resilience"

Timeouts, retries, circuit breakers, fallbacks, and recovery patterns.

View All Tags

The retry that made the outage worse

· 11 min read
Fault Lines Editorial
Fictional incidents. Exact technical vocabulary.
Architecture storyAdvancedA fictional production incident about retry ownership across an AI risk-decision path.

Kiteframe Pay (a fictional payment-risk infrastructure company) helps online marketplaces decide whether a card order should be approved, declined, or reviewed by a human. Merchants call Kiteframe during checkout, before they capture money or promise inventory to a shopper.

Most orders never need generative AI. Deterministic rules settle obvious cases in under 80 milliseconds. The difficult eight percent—new devices, unusual delivery patterns, sparse account histories—enter Aster, Kiteframe's AI-assisted risk analyst.

Aster was not a chatbot and could not approve a payment. It gathered approved evidence, used a language model to produce a typed risk recommendation with cited signals, and passed that recommendation to a deterministic merchant-policy engine. Without Aster, those ambiguous orders went to a manual-review queue. During a large sale, that queue could grow faster than Kiteframe's analysts could empty it.

At 02:13 on Tuesday, model latency rose sharply. Seven minutes later, Kiteframe's entire checkout success rate had fallen from 99.4% to 71%.

The model provider was recovering.

Kiteframe's retries would not let it.