2 posts tagged with "Cloud operations"

Slatebridge is a fictional cloud dispatch platform for independent home-services contractors — the heating, plumbing, and electrical companies with four to sixty technicians that arrive at your door in a small van. Roughly 2,400 fictional contractor companies and 19,000 technicians use it.

Slatebridge Desk is the browser app where a dispatcher builds tomorrow's route. Slatebridge Field is the phone app where a technician opens that route at shift start and marks each job done on the customer's driveway. Underneath, Slatebridge runs five cells, cell-01 through cell-05, with one shared entry layer in front of all five: Gatehouse.

Beatriz Quintero was the staff platform engineer who owned Gatehouse and Tenant Control, the service that creates contractors and decides where they live. Ivo Krastev was the site reliability engineer on call the morning this story happens. Wes Halloran ran support, so he learned the size of a problem before the dashboards did.

Slatebridge promised contractors one thing above all: from 06:30 local time onward, a technician can open the day's route in one tap. A technician who cannot open a route does not start driving.

Northstar Ledger (a fictional B2B commerce-infrastructure company) routes checkout and inventory updates for regional grocery chains. Its customers do not run one Northstar checkout page. They use Northstar behind their own mobile apps, self-checkout kiosks, and delivery sites.

At the Saturday peak, those systems send roughly 48,000 order operations per minute. A short outage creates queues. An incorrect recovery can create something worse: orders accepted through a path that the company no longer believes is authoritative.

Northstar stored checkout state in an Amazon Aurora PostgreSQL global database. The primary Region served writes. A secondary Region stayed ready for disaster recovery. The operating rule was strict: Northstar could advertise only the writer confirmed by Aurora's current topology.

Imani (the Staff Reliability Engineer responsible for Northstar's database-recovery control plane) had spent the previous six months reducing the time between an incident page and a safe recovery decision.

Her team built Relay, a bounded AI incident worker. Relay read approved telemetry, compared an incident with reviewed runbooks, and proposed a typed recovery plan. It could not promote a database or change traffic. A deterministic policy service checked every proposal. A human incident commander then approved the exact target and the maximum tolerated data loss before any effect was allowed.

This mattered because Relay was useful precisely where incidents were messy. It could gather replication lag, recent deployments, health probes, and runbook constraints in seconds. It could explain why one recovery target fitted the evidence better than another.

It could not make an ambiguous cloud operation unambiguous by thinking harder.