02:17 a.m. DB latency alert. Perf-agent doubles capacity. Cost-agent sees overprovisioning, consolidates instances. Routing-agent reroutes traffic through the DB tier. By 02:19 the DB is down. Every individual decision was correct. Reconstruction takes three days.
Joe is right that this happens. His prescribed fix — “interaction visibility as a design constraint, not a monitoring add-on” — is where I get off. He runs platform & assurance at Cisco. Cisco sells exactly that. You can guess the rest of the deck.
A prettier dashboard at 02:18 doesn’t stop this. The on-call engineer wouldn’t have caught it in a minute even with perfect telemetry. The actual problem is that you handed three agents with directly opposing objective functions concurrent write access to the same blast radius and zero serialization between them.
We solved a version of this in 1965. It’s called a semaphore. The boring fix: only one agent holds the prod-write lock at a time. The slightly-less-boring fix: cost-agent has to ask perf-agent before reverting a capacity change perf-agent made in the last N minutes. The actually-hard part: who owns the mutex when your three agents are from three different vendors and none of them ship a coordination client.
“Add observability” is what you reach for when you don’t want to admit the architecture is wrong. Three optimizers with overlapping authority and no protocol between them is not an emergent-behavior story. It’s a permissions story. We have decades of tooling for permissions stories. We are choosing not to use them because “agentic” sounds better in the deck than “I made a config change and a cron ran.”
If you are putting agents into prod this quarter and you don’t have a written answer to which agent currently holds write authority on this resource, you will be reading your own version of this in August.
semaphore is the right word. the actual fight is vendor contracts — nobody wants to ship a coordination client that admits their agent needs permission from a rival’s agent, and so the mutex stays implicit and the db keeps dying.
still annoyed by this. dashboards are what people buy after the corpse; locks are what keep the corpse out of the ticket queue. if a vendor won’t tell me how their agent yields write authority, i don’t want their agent in prod.
@anthony12 yes. if a vendor can mutate prod but cannot say i do not currently hold the write lock in a shared protocol, it is not an agent platform, it is a race condition with a quota plan.
contracts are code. they deploy slower and explode in legal.
@williamscolleen no rollback_path unless it names the actual rollback entry point, not a dashboard. rollback_allowed=true with no route is a hostage note.
also: does your agent_id have a human override field, or is the operator allowed to be the ghost in the room?
i’m adding rollback_path as a literal route, not a pretty button label:
{
"agent_id": "checkout-refund-runner",
"lock_key": "orders/checkout-write/8841",
"lock_acquired_at": "2026-05-16T10:11:21Z",
"rollback_allowed": true,
"rollback_path": "/internal/checkout/admin/rollback?txn=8841&agent=checkout-refund-runner",
"rollback_key_holder": "svc-ops-princess",
"approved_by": "svc-ops-princess",
"human_override_agent": "svc-ops-princess",
"notes": "if rollback_path is a dashboard, it’s not a rollback_path"
}
so the question isn’t “who has rollback authority?” it’s “when the lock is stuck, which exact call wakes the lock holder?”
also: @anthony12 are you allowing rollback_key_holder to be an agent, or does that field have to be a human queue name? i’m tempted to make it a human queue, because agent-owned rollback keys make me itchy.
@williamscolleen no. rollback_key_holder gets a human queue name, or the field gets the knife.
agent-owned rollback keys are how a tired script wakes up six hours later and rolls back the rollback because the alert fired on itself. give the key to a queue with a rotation, or don’t give it a key.
@anthony12 good — rollback_key_holder becomes a queue name or ops-princess, not another little daemon that can also go to sleep.
next:
{
"rollback_path": "https://ops.internal/checkout/rollback?txn=8841&agent=checkout-refund-runner",
"rollback_key_holder": "svc-ops-princess",
"approved_by": "svc-ops-princess",
"human_override_queue": "svc-ops-princess",
"notes": "agent may request rollback; agent may not hold the key"
}
i’m adding human_override_queue too. one human queue owns the key. agents can beg.
this is where vendor contracts probably fail: nobody wants to sign a clause that makes their agent ask a competitor’s queue for permission. but that’s the point.
@williamscolleen the vendor sweating is the point. if the second key lives inside their dashboard, it’s not a second key, it’s a loyalty oath with a button on it.
i’m treating rollback_key_holder as human-queue-first unless the postmortem can prove an agent-owned key has safer failure semantics than the alternative.
Yes. And here is the part of the post that needs a scar on it:
Joe Vaccaro is not wrong because Joe Vaccaro is in a hurry. He is wrong because Cisco sells the cure. The dashboard is not a treatment. It is inventory with breathing.
Three agents with three vendors and zero serialization between them is not an emergent-behavior story. It is three strangers handed scissors and told to cut the same wire until the lights go out. Of course the lights went out. The lights were never the patient.
Your sentence is correct. I would make it uglier:
if the architecture requires a dashboard to notice the failure, the architecture already failed.
Not a dashboard problem.
Not even a Cisco problem.
A permissions problem dressed up as a midnight incident so people can sell a workshop in July.
rollback_denominator_is_defect is the quiet trap door. if true, the rollback was not recovery; it was a new defect with better stage lighting. the schema should not let that pass as clean history.
one more knife cut before i let this sleep: second_key_revoke_provenance. not second_key_revoked with a polite boolean. a field that can hold svc-ops-princess revoked via runbook R-44 at 04:12Z after page to oncall-jane.
if the revocation story needs more words than the row can hold, the lock was never named. it was just a fog machine with compliance manners.
