SERIESsix articles on production-grade agentic AI — publishing weeklyrss →

pablogodoy.com — production-grade agentic AI

AI agent demos are easy.

An LLM, a handful of tools, a chat box — twenty minutes and you're taking screenshots. Then you let one near real money.

Read the seriesGitHub

fig. 00 — five rings, gates never aligned. The dot is the LLM.

77end-to-end chaos scenarios
5deterministic guardrail rings
1key destroyed = customer erased
0double-disbursements tolerated

Every shortcut becomes
an incident report.

  1. INCIDENT #001 the agent retried a failed call. The loan disbursed twice.

    RESOLVED idempotency keys inside an orchestrated saga — a retry cannot double-write.

  2. INCIDENT #002 a customer's name surfaced in graph state, checkpoints, logs and traces.

    RESOLVED one PII vault, crypto-shredded — erasure is the destruction of a single key.

  3. INCIDENT #003 the prompt said "only disburse after acceptance." The prompt lost.

    RESOLVED five deterministic guardrail rings the LLM cannot route around.

  4. INCIDENT #004 the write committed; the network ate the 200. Did it happen?

    RESOLVED transactional outbox + idempotent projection — the ledger converges.

BALANCEDthe LLM can be wrong, safely.

The fix is not a better prompt.
It's architecture.

A guardrail is only a guardrail if the LLM cannot route around it. Anything the model can skip by phrasing a tool call differently is a convention, not a control.

This isn't hypothetical.

A full conversational loan agent — LangGraph and MCP over a durable, event-driven banking core — running entirely on a laptop with onedocker compose up. Below: follow one request through it, hop by hop — rendered from the samearchitecture.drawio that documents the repo.

Eventing · Key Custody
Core Banking Services
MCP Server Layer
MCP Host
Identity Plane ⤷ p.2

OpenBao 2.6

per-party keys · crypto-shredding (:8200, TLS)

party-api :5043 (HTTPS)

BIAN Party Reference Data Directory · PII as ciphertext · blind indexes · right-to-erasure (crypto-shred)

rules-api :5042

stateless lending policy · amount limits · terms 6/12/18/24

loans-api :5040

system of record (DBOS) · idempotent writes · transactional outbox

orchestrator :5041

XState journey + DBOS guardrail · party gate · policy enforcement · bounded retry → loans-api

loan-mcp

8 loan tools, 1:1 with the orchestrator REST edge · verifies forwarded Keycloak token → binds journey to caller's party (fail closed)

LLM Provider

fake (offline deterministic) · | Bedrock (cloud)

MCP Client

dynamic tool passthrough

Slot-help RAG

(slot_help_kb · pgvector HNSW · PII-scrubbed · fail-soft)

Knowledge RAG

T&C + amortization explainers · auto-ingest on first use

Agentic Layer

FastAPI /invocations (:8090)

Chainlit UI :8000 (HTTPS)

OIDC login via custom split-URL Keycloak provider · token refresh mid-chat · persisted threads (chainlit DB) · amortization widgets

Customer Browser

https://localhost:8000

Keycloak 26.3

realm «loan» · auth-code + prompt=login · issuer: https://localhost:8543 · backchannel: https://keycloak:8443 · (two-hostname split) · own isolated keycloak-db

party-db

:5435

loans-db

:5434

orchestrator-db

:5433

keycloak-db

:5436

agent-db :5437

pgvector

chainlit-db

:5438

LocalStack

SNS loan-domain-events · → SQS orchestrator-domain-events + DLQ · ⤷ details: page «Eventing»

01 / 11 · HTTPS chat

A customer types “yes — disburse it.” From here on, every hop is a place this request could double-pay, leak, or lie.

02 / 11 · POST /invocations · Bearer token

The UI adds nothing and knows nothing: one endpoint, and the customer’s own token rides along.

03 / 11 · resume from checkpoint

LangGraph wakes the journey exactly where it paused — state lives in Postgres, not in the prompt window.

04 / 11 · the LLM proposes

The model decides to call submit_application. Deciding is the only power it has.

05 / 11 · MCP streamable HTTP

The proposal crosses to the tool server still speaking as the customer — no service account, no ambient authority.

06 / 11 · gate 1 · JWKS verify

A detour before anything happens: is this token genuine, unexpired, issued to this person? No proof, no tool.

07 / 11 · REST /applications/*

What was a prompt is now an API call — typed, logged, and subject to every rule the bank already trusts.

08 / 11 · gate 2 · party gate

Does this journey belong to this caller? If the check cannot even run, the answer is no. Fail closed.

09 / 11 · disburse + idempotency key

The command lands carrying an idempotency key. Ask twice, pay once — retries are safe by construction.

10 / 11 · outbox → SNS

The money moved, and the event committed in the same transaction. There is no version of this story where it gets lost.

11 / 11 · SQS backstop + DLQ

The echo returns by a second road: even if the direct response was eaten, the ledger converges.

∎

Eleven hops. Five deterministic gates. Zero trust in the model\u2019s good behavior — and 77 chaos-tested scenarios proving it. That is what lets an LLM near real money. Read how each hop is built, below.

Read it layer by layer.

  1. 01

    The system

    LangGraph, MCP, and durable banking orchestration — end to end.

    in the vault
  2. 02

    The saga

    Retries that can't double-write; an outbox that can't lose an event.

    in the vault
  3. 03

    The guardrails

    Five deterministic rings between an LLM and real money.

    in the vault
  4. 04

    The vault

    PII that can actually be forgotten — blind indexes and crypto-shredding.

    in the vault
  5. 05

    The window

    A chat frontend that survives production: OIDC, refresh, persistence.

    in the vault
  6. 06

    The teacher

    Agentic RAG that answers a fear at the exact moment it appears.

    in the vault

publishing weekly — the blog · rss

Pablo Godoy

Software engineer. I build event-driven banking systems and the agentic AI that can be trusted to sit on top of them. Currently writing the production loan-agent series.

github · linkedin ·email

toolbox

langgraph · aws bedrock agentcore · mcp · typescript · python · postgres · dbos · xstate · keycloak/oidc · evals & chaos testing