You can't un-send a double charge
What blows agents up in production usually isn't bad reasoning — it's side effects. On retry: double post, double charge, double email. Making every external action safe-to-retry did more than any model upgrade.
Once you actually leave an agent alone in production, the failure that gets you out of bed rarely comes from where you expect. The model 'reasoning wrong' is rare; the real trouble is side effects. The agent completes a step, the network hiccups, the harness retries — and the same external action fires a second time. A double post. A double email. A customer charged twice. The model may have reasoned perfectly, and two invoices still went out.
The asymmetry here is the whole point: you can take back a bad answer, but you can't take back a charge that already went out. Un-thinking a thought is free; un-sending an action is impossible. That's why what makes an agent leave-able isn't a smarter model — it's every external action being safe-to-retry.
The problem is repetition, not reasoning.
An autonomous agent means automatic retry. Strip the retry and the agent dies on the first network blip; keep it and every shipped step risks running again. So the center of the reliability problem isn't the model's intelligence — it's the idempotency of your actions. One rule changes everything: any call that touches the outside world must behave as if it ran once, even when it runs twice with the same input.
// Fragile: a retry means a second charge
await stripe.charges.create({ amount, customer });
// Safe: same key → one charge, retry is harmless
await stripe.charges.create(
{ amount, customer },
{ idempotencyKey: `charge:${orderId}` }, // derive from the work
);The key must not be a random UUID — derive it from the identity of the work (order id, message id, booking id). That way the retry carries the same key and the provider says 'already did this.' Emails, webhooks, DB inserts, third-party APIs: all guarded the same way — unique constraints, upserts, a dedup table.
What I actually do.
- ▸Run every outward-facing action through one question — 'what happens if this fires twice?' If the answer is 'bad,' it needs an idempotency key.
- ▸Derive the key from the work's identity, never generate it randomly.
- ▸Collect irreversible actions (charges, sends) into a single, explicit step at the very end of the reasoning.
- ▸Keep a dedup layer that marks each action 'done' so it can't be triggered again.
Doing this is far less exciting than waiting for a new model to drop. But it has done more for my production reliability than the last several model upgrades combined. A smarter model makes the work faster; safe-to-retry actions make the work leave-able.
“You can pretend you never thought a wrong answer. You can't take back a charge that already went out. That difference is the entire question of whether an agent ships.”