SERIESsix articles on production-grade agentic AI — publishing weeklyrss →

pablogodoy.com — production-grade agentic AI

AI agent demos are easy.

An LLM, a handful of tools, a chat box — twenty minutes and you're taking screenshots. Then you let one near real money.

Read the seriesGitHub

fig. 00 — five rings, gates never aligned. The dot is the LLM.

77end-to-end chaos scenarios
5deterministic guardrail rings
1key destroyed = customer erased
0double-disbursements tolerated

Every shortcut becomes
an incident report.

  1. INCIDENT #001 the agent retried a failed call. The loan disbursed twice.

    RESOLVED idempotency keys inside an orchestrated saga — a retry cannot double-write.

  2. INCIDENT #002 a customer's name surfaced in graph state, checkpoints, logs and traces.

    RESOLVED one PII vault, crypto-shredded — erasure is the destruction of a single key.

  3. INCIDENT #003 the prompt said "only disburse after acceptance." The prompt lost.

    RESOLVED five deterministic guardrail rings the LLM cannot route around.

  4. INCIDENT #004 the write committed; the network ate the 200. Did it happen?

    RESOLVED transactional outbox + idempotent projection — the ledger converges.

BALANCEDthe LLM can be wrong, safely.

The fix is not a better prompt.
It's architecture.

A guardrail is only a guardrail if the LLM cannot route around it. Anything the model can skip by phrasing a tool call differently is a convention, not a control.

This isn't hypothetical.

A full conversational loan agent — LangGraph and MCP over a durable, event-driven banking core — running entirely on a laptop with onedocker compose up. Below: follow one request through it, hop by hop — rendered from the samearchitecture.drawio that documents the repo.

Eventing · Key Custody
Core Banking Services
MCP Server Layer
MCP Host
Y7yOPG4R-pF25_CY23qE-2

OpenBao 2.6

per-party keys · crypto-shredding (:8200, TLS)

party-api :5043 (HTTPS)

BIAN Party Reference Data Directory · PII as ciphertext · blind indexes · right-to-erasure (crypto-shred)

rules-api :5042

stateless lending policy · amount limits · terms 6/12/18/24

loans-api :5040

system of record (DBOS) · idempotent writes · transactional outbox

orchestrator :5041

XState journey + DBOS guardrail · party gate · policy enforcement · bounded retry → loans-api

loan-mcp

8 loan tools, 1:1 with the orchestrator REST edge · verifies forwarded Keycloak token → binds journey to caller's party (fail closed)

LLM Provider

fake (offline deterministic) · | Bedrock (cloud)

MCP Client

dynamic tool passthrough

Slot-help RAG

(slot_help_kb · pgvector HNSW · PII-scrubbed · fail-soft)

Knowledge RAG

T&C + amortization explainers · auto-ingest on first use

Agentic Layer

FastAPI /invocations (:8090)

Chainlit UI :8000 (HTTPS)

OIDC login via custom split-URL Keycloak provider · token refresh mid-chat · persisted threads (chainlit DB) · amortization widgets

Customer Browser

https://localhost:8000

Keycloak 26.3

realm «loan» · auth-code + prompt=login · issuer: https://localhost:8543 · backchannel: https://keycloak:8443 · (two-hostname split) · own isolated keycloak-db

party-db

:5435

loans-db

:5434

orchestrator-db

:5433

keycloak-db

:5436

agent-db :5437

pgvector

chainlit-db

:5438

01 / 11 · HTTPS chat

A customer asks about a loan. Real browser, real OIDC session — not a demo user.

02 / 11 · POST /invocations · Bearer token

The UI stays domain-blind: one endpoint, and the user’s token travels with the request.

03 / 11 · LangGraph checkpoint

The flow resumes exactly where it paused — state lives in Postgres, not in the prompt.

04 / 11 · decision: submit_application

The LLM proposes a tool call. Proposing is all it can do.

05 / 11 · MCP streamable HTTP

The call crosses to the tool server — still carrying the user’s token, never a service account.

06 / 11 · JWKS verify

Detour: no valid token, no tool call. The tool layer re-verifies identity itself.

07 / 11 · REST /applications/*

The prompt has become an API call — typed, versioned, enforceable.

08 / 11 · party gate — fail closed

Does this journey belong to this caller? If the check cannot run, the answer is no.

09 / 11 · command + idempotency key

The disburse command carries an idempotency key — a retry cannot double-write.

10 / 11 · outbox → SNS

The event commits in the same transaction as the write. It cannot be lost.

11 / 11 · SQS backstop + DLQ

And it comes back: the projection converges even if the direct response was eaten.

∎

One request, eleven guarded hops — proven by 77 chaos-tested e2e scenarios. The whole bank runs offline: docker compose up.

Read it layer by layer.

  1. 01

    The system

    LangGraph, MCP, and durable banking orchestration — end to end.

    in the vault
  2. 02

    The saga

    Retries that can't double-write; an outbox that can't lose an event.

    in the vault
  3. 03

    The guardrails

    Five deterministic rings between an LLM and real money.

    in the vault
  4. 04

    The vault

    PII that can actually be forgotten — blind indexes and crypto-shredding.

    in the vault
  5. 05

    The window

    A chat frontend that survives production: OIDC, refresh, persistence.

    in the vault
  6. 06

    The teacher

    Agentic RAG that answers a fear at the exact moment it appears.

    in the vault

publishing weekly — the blog · rss

Pablo Godoy

Software engineer. I build event-driven banking systems and the agentic AI that can be trusted to sit on top of them. Currently writing the production loan-agent series.

github · linkedin ·email

toolbox

langgraph · aws bedrock agentcore · mcp · typescript · python · postgres · dbos · xstate · keycloak/oidc · evals & chaos testing