The Saga Pattern Under an AI Agent — Orchestration, Idempotency, and the Transactional Outbox

A loan journey is a saga: a sequence of commands across services, each individually transactional, none jointly so. Here’s how an orchestrated saga stays correct when the network eats a response — retries that can’t double-write, an outbox that can’t lose an event, and a consumer that can’t apply one twice.

There is exactly one failure that separates toy distributed systems from production ones, and it fits in a sentence:

The write committed, and the response was lost.

The orchestrator calls loans-api to disburse. loans-api commits. The connection resets before the 200 arrives. Did the disbursement happen? The caller cannot know. Retry blindly and you disburse twice; never retry and every network blip strands a customer mid-journey.

Under an AI agent this failure is worse, because the retry pressure is conversational — the customer says “try again,” the LLM happily re-calls the tool, and now your dedup story is being exercised by a language model. This article walks the full machinery that makes that safe: an orchestration-based saga, idempotency keys done properly, a transactional outbox, and an idempotent projection — with every piece proven by chaos tests.

The whole architecture is that picture: the reply can vanish, so there are two independent roads from the system of record back to durable state — and both are safe to travel twice.

Orchestration, not choreography

The loan journey — create → simulate → accept → disburse, with cancellation paths — is driven by a central command service in the orchestrator. Each step is a synchronous command against loans-api; a pure XState machine answers “is this step legal from here”; compensations (cancel the old acceptance before re-simulating; cancel simulation and acceptance when the journey is abandoned) are explicit code in the orchestrator, not emergent behavior of event subscribers.

The choreography alternative — services reacting to each other’s events — was rejected for a reason that matters double under an agent: you cannot explain an emergent saga to a customer. The agent needs to answer “where are we, and what can happen next?” on every turn; a central Control Record with currentState and HATEOAS-style availableActions is that answer. One deliberately vestigial detail: disbursed is terminal and has no compensation — the cancel path for it exists only for symmetry and is never legal. Money that left is a new business process (collections), not an undo.

Idempotency: the key is the contract

Every logical command carries an idempotency key, and the scheme of those keys encodes business intent:

// Keys are deterministic PER ENTITY so a re-simulation gets a FRESH key (the old constant
// `${id}:simulation` key made loans-api dedupe a re-sim and return the stale simulation):
//   - simulation:   `${id}:simulation:${nonce}` — a fresh nonce per simulate COMMAND.
//   - acceptance:   `${id}:acceptance:${simulationId}` — one acceptance per active simulation.
//   - disbursement: `${id}:disbursement` — exactly one disbursement per journey (terminal).
export function simulationIdempotencyKey(controlRecordId: string): string {
  return `${controlRecordId}:simulation:${newId()}`;
}

Read those three lines again — they are three different answers to “what does the same command mean?” A retry of one simulate replays one nonce. A new simulate mints a new nonce (the old constant-key version deduped re-simulations into stale quotes — a real bug). And a journey gets exactly one disbursement, ever.

On the loans-api side, the envelope is brutally simple, and the simplicity is the point:

export async function runIdempotentWrite<T>(
  uow: UnitOfWork, idempotencyKey: string, responseSchema: ZodType<T>,
  produce: (tx: LoansTx) => Promise<T>,
): Promise<T> {
  return uow.transaction(async (tx) => {
    const existing = await tx.findIdempotentResponse(idempotencyKey);
    if (existing !== null && existing !== undefined) {
      return responseSchema.parse(existing);
    }
    const result = await produce(tx);
    await tx.saveIdempotentResponse(idempotencyKey, result);
    return result;
  });
}

The recorded response is saved in the same transaction as the domain write (idempotency_keys.key is the primary key — the unique constraint is the correctness proof). A replayed key returns the recorded response. There is no window where the write exists but the receipt doesn’t.

Retry: bounded, classified, and invisible to the caller

With idempotency underneath, retrying becomes safe — but only worth doing for the right failures:

// Transient = a fault a RETRY could plausibly clear: a 5xx UpstreamError or a bare
// transport failure (connection reset / timeout). Terminal = any 4xx DomainError
// (validation, not-found, illegal transition): retrying only burns patience.

The schedule is five attempts, 1s base, ×2 growth — roughly 15 seconds of total backoff — with additive jitter. Additive matters: full jitter can collapse a delay toward zero, and this system’s chaos test asserts elapsed > 12s under a persistent fault, proving the schedule actually ran. Jitter that can shrink the floor would make the proof flaky; jitter that only adds decorrelates concurrent retriers without touching it.

The composition is the part worth copying:

export function createRetryingLoansClient(inner: LoansClient, policy: RetryPolicy): LoansClient {
  return {
    simulate: (request) => policy.execute(() => inner.simulate(request)),
    accept: (request) => policy.execute(() => inner.accept(request)),
    disburse: (request) => policy.execute(() => inner.disburse(request)),
    /* …cancel ops… */
  };
}

A plain decorator at the composition root. The command service depends on LoansClient and does not know retries exist; the retry module knows nothing about loan semantics. One secret per module — when the schedule changes, one file changes; unit tests inject fake timers and fake randomness, production injects nothing.

The precondition is stated where it can’t be missed:

// PRECONDITION: `op` MUST be idempotent. A retry may replay a call whose earlier
// attempt already committed downstream but whose response was lost (the ambiguous
// request-landed-response-lost failure). Every loans-api call satisfies this.

The transactional outbox: the event cannot not exist

Retries fix the caller’s view. But the orchestrator’s projection also needs the event — and publishing to a broker from inside a request handler is how events get lost. The outbox pattern closes that hole: loans-api writes the domain row and the event row in one transaction:

CREATE TABLE IF NOT EXISTS outbox (
  id TEXT PRIMARY KEY,          -- the event's UUIDv7 = dedup/order key, end to end
  event_type TEXT NOT NULL,
  control_record_id TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'PENDING',   -- PENDING -> NOTIFIED
  attempts INTEGER NOT NULL DEFAULT 0,
  published_at TIMESTAMPTZ NULL
);
-- Partial index keeps the relay poll O(pending) despite an ever-growing NOTIFIED backlog:
CREATE INDEX IF NOT EXISTS idx_outbox_pending ON outbox (id) WHERE status = 'PENDING';

One subtle move: the event id is minted first and embedded in both the HTTP response and the outbox row. The synchronous path and the async path carry the same UUIDv7, so the projection can dedup across both with a single comparison.

A DBOS-durable relay drains the table every second — claim with FOR UPDATE SKIP LOCKED, publish to SNS inside a durable step, then mark NOTIFIED guarded on status = 'PENDING'. A crash between publish and mark? The row republishes next tick. That’s at-least-once by design, which is fine, because…

…the consumer is idempotent three ways

SNS fans out to an SQS queue (three failed receives → DLQ); the orchestrator’s consumer applies events through the same applier the synchronous path uses:

// 1. Dedup / ordering guard. eventIds are UUIDv7 (monotonic, string-sortable): a stale or
//    already-applied event is a no-op.
if (current.lastAppliedEventId !== null && event.eventId <= current.lastAppliedEventId) {
  return current;
}
// 3. Machine-legality gate: an out-of-order backstop event … is skipped rather than
//    corrupting state.
if (!canFire(current.currentState, { type: machineEvent })) {
  return current;
}

…and the final write is a compare-and-swap on a monotonic version. Not on state — a re-simulation is a simulated → simulated self-loop, and a state-keyed CAS matches stale rows. That was a live bug; the monotonic version fixed it.

So the queue is a backstop, not a source of truth: state is durable in loans-api regardless, the synchronous path projects immediately, and nearly every queued event dedups to a no-op. The honest cost, called out in the code: the backstop carries every domain event, so it must drain full production volume just to stay at zero backlog — which is why it’s competing SQS consumers rather than the original one-tick-per-second DBOS scheduled workflow that capped throughput at ~10 messages/s.

The failure, replayed end to end

Here’s the full sequence, actor by actor:

Trace the scenario: simulate, attempt 1, commit, response lost.

1. Key CR-1:simulation:<nonce> minted once; attempt 1 commits row + outbox event E + recorded response, then the 200 dies on the wire.
2. The retry client classifies the transport error transient, sleeps ~1s, replays the same key.
3. loans-api finds the key → returns the recorded response. No second simulation, no second event.
4. The orchestrator projects E synchronously: version CAS 0→1, state simulated.
5. Meanwhile the relay publishes E → SNS → SQS; the consumer sees E ≤ lastAppliedEventId → no-op; message deleted.

Final state: exactly one simulation, one Control Record at version 1 — indistinguishable from the run where nothing failed. The e2e suite forces this exact path with Toxiproxy and asserts both the single simulation and the elapsed-time floor proving the backoff ran.

Conclusion

Nothing here is exotic — saga orchestration, idempotency keys, an outbox, an idempotent consumer are all twenty-year-old ideas. What makes the combination production-grade is that each mechanism assumes the others will be exercised: the retry assumes responses get lost, the keys assume retries happen, the outbox assumes the process dies mid-publish, the consumer assumes the queue delivers twice and out of order. And the chaos tests assume nothing — they inject the faults and measure.

If you keep one sentence: make every command mean something exact (the key scheme), make every write self-announcing (the outbox), and make every apply harmless to repeat (the idempotent projection) — then retries stop being scary and become boring. Boring is the goal.