Build a Production-Grade Conversational Loan Agent — LangGraph, MCP, and Durable Banking Orchestration

From MCP tool design to a durable XState + DBOS guardrail, end-to-end identity with Keycloak, crypto-shredded PII, and an eventing backstop — everything you need to let an AI agent near real money. And it all runs fully offline on your laptop.

AI agent demos are easy. An LLM, a handful of tools, a chat box — twenty minutes and you’re taking screenshots.

Then you point the same architecture at a banking workflow, and every shortcut turns into an incident report:

• The agent retries a failed call and disburses the loan twice.
• The LLM gets creative and calls disburse before the customer accepted the terms.
• A network blip mid-conversation and the application state evaporates.
• The customer’s name and email end up in your logs, your traces, and your vector store — right before they invoke their right to erasure.
• Anyone who can reach the API can operate on anyone else’s loan, because “the agent sends a customer id” is the entire identity model.

The problem isn’t the LLM. The problem is treating a probabilistic text generator as if it were a trustworthy client. It isn’t one. It’s a brilliant, unreliable intern — and the system around it has to be built the way banks have always built systems around unreliable clients: with hard state machines, idempotency, verified identity, and an audit trail.

That’s exactly what this guide builds. Not a theory-heavy overview — a real, running system you can clone, boot with one docker compose up, and break on purpose.

To run the demo and every experiment in this article, check out the GitHub repo for full instructions.

What are we going to build?

By the end of this guide, you’ll have a fully functional conversational loan-acquisition system: a customer chats with an agent, picks a loan product, gets a simulation with an interactive amortization chart, accepts the terms, and receives a (demo) disbursement — with every step enforced, persisted, verified, and traceable.

The customer experience looks like an AI chat. Underneath, it’s a small bank:

Here’s what you’ll walk away with:

• A pattern for putting a deterministic guardrail between an LLM and anything irreversible
• An MCP tool surface designed for agents — docstrings as prompts, errors as data, ordering enforced server-side
• Retry + idempotency done properly, including the failure mode most systems ignore
• A transactional outbox → SNS → SQS projection backstop, with a DLQ
• End-to-end identity: OIDC login, token forwarding, and independent re-verification at three hops
• Right-to-erasure by crypto-shredding — PII that can actually be forgotten
• A fully offline stack — deterministic fake LLM, LocalStack instead of AWS — that still runs 77 end-to-end scenarios, chaos included

The core principle: the LLM proposes, the machine disposes

Before any code, one design decision drives everything else in this system:

The LLM never decides what is legal. It only proposes. A pure, deterministic state machine — persisted durably — is the single authority on what may happen next.

Every “agent gone rogue” story is a violation of this principle. If your agent can call disburse and the only thing preventing an out-of-order disbursement is the prompt — you don’t have a workflow, you have a suggestion.

In this system the loan lifecycle is an explicit XState machine:

export const LoanStateSchema = z.enum([
  'selecting_product',
  'offering_shown',
  'collecting_inputs',
  'simulated',
  'terms_accepted',
  'disbursed',
  'cancelled',
  'failed',
]);

// The loan-acquisition lifecycle as a PURE, deterministic XState machine. It is the
// guardrail: given the current durable state and a candidate event (proposed by the
// non-deterministic LLM, or projected from a loans-api domain event), it answers
// "is this transition legal?" and "what is the next state?". It performs NO side
// effects — everything depends on the machine; the machine depends on nothing.

The machine is pure — no I/O, no context, no side effects. The orchestrator persists the current state durably (DBOS on Postgres), and every command the agent proposes is checked against the machine before anything executes. An out-of-order call — say, disburse_loan while the state is still simulated — doesn’t corrupt anything. It comes back as a clean 409 STATE_TRANSITION error that the agent can read, explain, and recover from conversationally.

That one boundary changes the whole risk profile: the LLM can be as creative as it wants, because creativity bounces off the guardrail.

The MCP tool surface: designing tools for an unreliable caller

The agent reaches the banking core exclusively through loan-mcp — a FastMCP server exposing 8 tools that map 1:1 to the orchestrator’s REST edge. This layer looks thin, but it’s where most of the agent-facing design lives.

mcp = FastMCP("loan-acquisition", host=MCP_HOST, port=MCP_PORT)

@mcp.tool()
def get_loan_products() -> dict[str, Any]:
    """List the loan products the customer can choose from (the catalog).

    Call this FIRST, at the very start of a loan conversation, to present the
    available products (e.g. personal / auto / mortgage) as discrete choices. Each
    product carries the limits and installments the later steps must respect.

    Returns:
        {"products": [ <product>, ... ]} on success, each product shaped like
        {"id", "label", "currency", "minAmountMinor", "maxAmountMinor",
         "allowedInstallments": list[int], "baseRateBps", "disclosure"}
        (amounts in minor units, e.g. cents), or {"error": <RFC 9457 problem>} if
        the rules service is unavailable.
    """
    return _run_rules("get_loan_products",
                      lambda: {"products": _rules_client.get_products()})

Three deliberate choices are packed in here:

Docstrings are prompts. The LLM reads these to decide when and how to call each tool. “Call this FIRST” isn’t documentation flavor — it’s behavioral steering that costs zero tokens of system prompt. Every one of the 8 tools carries ordering hints, argument semantics, and the exact response shape.

Errors are data, not exceptions. A failed call returns {"error": <RFC 9457 problem>} instead of raising. An exception dies somewhere in the MCP transport; a structured error reaches the model, which can then tell the customer “that amount is above the limit for this product” instead of stalling. The orchestrator’s 409 guardrail rejections ride the same channel.

Money is integers. Every amount is minor units (amount_minor, cents) end to end. LLMs are bad at floating point; ledgers are worse at it.

The module docstring states the decomposition contract explicitly — this file knows nothing about HTTP:

"""
Secret this module hides: the MCP tool surface — tool names, docstrings, typed
signatures, and how an OrchestratorError becomes a usable {"error": {...}} the
agent can read. It knows NOTHING about httpx, paths, or the HTTP contract;
those are orchestrator_client.py's and rules_client.py's secrets. The only
reason to change this file is a change to the agent-facing tool shape.
"""

When the tool shape changes, one file changes. When the HTTP contract changes, a different file changes. That’s the whole game.

The end-to-end flow

One full journey, numbered:

1. Login — The customer signs in through Keycloak (a real authorization-code flow — more on this below) and opens the chat.
2. Chat turn — The UI POSTs to the agent’s /invocations endpoint with the customer’s access token.
3. Catalog — The agent calls get_loan_products → get_offering, presenting products and terms as discrete choices.
4. Slot filling — The LangGraph flow collects amount and installments, pre-validating against the offering’s limits so out-of-policy requests get caught conversationally.
5. Simulation — simulate_loan mints the application (the Control Record) via the orchestrator; the UI renders the payment plan plus an interactive amortization chart.
6. Guardrailed progress — accept_terms, then disburse_loan — each command checked against the XState machine, executed against loans-api with retry + idempotency, then projected back into durable state.
7. Events — loans-api’s outbox publishes domain events to SNS; the orchestrator’s SQS consumer converges the projection even if a direct response was lost.

Note: the LLM may loop between steps 3–6 as the conversation meanders — customers change amounts, ask questions, go back. That’s fine because ordering is enforced server-side. The conversation is free; the ledger is not.

Surviving failure: bounded retries on top of idempotency

Here’s the failure mode that separates toy agents from production systems: the orchestrator calls loans-api, loans-api commits the write, and the response is lost — connection reset after commit. Did the disbursement happen? The caller cannot know.

If you retry blindly, you double-disburse. If you never retry, every network blip strands a customer. The answer is a pair of mechanisms that only work together:

1. Every logical command carries an idempotency key, and loans-api dedupes on it:

• The key is built once per logical command — a retry replays the same key.
• loans-api’s runIdempotentWrite records the response under that key inside the same transaction as the domain write.
• A replayed key returns the recorded response. Never a second write.

2. The orchestrator retries only what’s worth retrying, on a bounded schedule:

// Transient = a fault a RETRY could plausibly clear: a 5xx UpstreamError or a bare
// transport failure (connection reset / timeout). Terminal = any 4xx DomainError
// (validation, not-found, illegal transition): retrying only burns patience.
export function isTransient(error: unknown): boolean {
  if (isDomainError(error)) {
    return !(error.status >= 400 && error.status < 500);
  }
  return true;
}

// maxAttempts 5, 1s base, ×2 growth: retries at ~1s, 2s, 4s, 8s → ~15s of backoff.
// Jitter is ADDITIVE (layered on top, never subtracted): it decorrelates concurrent
// retriers but cannot shrink the total below the ~15s floor.
export const DEFAULT_RETRY_SCHEDULE: RetrySchedule = {
  maxAttempts: 5,
  baseDelayMs: 1_000,
  backoffFactor: 2,
  jitterMs: 250,
};

The comment in the real code says it plainly:

// PRECONDITION: `op` MUST be idempotent. A retry may replay a call whose earlier
// attempt already committed downstream but whose response was lost (the ambiguous
// request-landed-response-lost failure). Every loans-api call satisfies this.

Two details worth stealing:

• The retry policy is a plain injected decorator (retrying-loans-client wraps the loans client). The command service doesn’t know retries exist. Unit tests inject fake timers and fake randomness; production injects nothing and gets the wall clock.
• The chaos tests prove the retries ran. The e2e suite uses Toxiproxy to inject a persistent fault and then asserts elapsed > 12s — if the backoff schedule silently stopped running, the test fails. That’s also why the jitter is additive rather than “full jitter”: full jitter can collapse a delay toward zero and would make that proof flaky.

The eventing backstop: outbox → SNS → SQS

The synchronous path can still lose a response after all retries. For that, the system keeps a second, asynchronous road to consistency:

The sequence has three phases:

1. Command — loans-api writes the domain change and an outbox row in one transaction. There is no window where the state changed but the event doesn’t exist.
2. Async relay — a DBOS-durable poller publishes unpublished outbox rows to an SNS topic (loan-domain-events) and marks them published. At-least-once, by design.
3. Projection backstop — an SQS queue subscribed to the topic feeds the orchestrator’s consumer, which applies events to its projection idempotently per event id. Three failed receives and the message redrives to a DLQ instead of poisoning the queue.

Backstop, not source of truth. State is durable in loans-api regardless. The event path exists so the orchestrator’s view converges even when a direct response was lost — and duplicates are absorbed by idempotency on both sides.

Locally, SNS and SQS are LocalStack — the offline stack exercises the real AWS SDK code paths, queue redrive policy included, without an AWS account.

Identity end to end: the agent is just another untrusted client

Most agent demos handle identity like this: the UI tells the agent who the user is, and everyone downstream believes it. In a loan flow, that’s a vulnerability, not a shortcut.

This system treats the access token as the only identity carrier, and re-verifies it independently at every hop that matters:

Phase 1 — Login. The Chainlit UI runs a real authorization-code flow against Keycloak with prompt=login (logout must actually log out — an SSO cookie that silently waves you back in is wrong for a loan flow). One subtlety worth knowing: the browser reaches Keycloak at https://localhost:8543 while backends reach it in-network at https://keycloak:8443 — the classic two-hostname split. The token’s issuer is pinned to the frontend URL; the JWKS is fetched in-network. Chainlit’s stock provider assumes one base URL for both, so the UI registers a small custom provider that splits them.

Phase 2 — Every chat turn. The UI forwards the access token to the agent; the agent forwards it to loan-mcp; and then the verification cascade begins:

def verify(self, token: str) -> VerifiedPrincipal:
    """Prove the token authentic, then return its principal; raise TokenError otherwise.

    Verifies (in order): a signing key exists in JWKS for the token's kid; RS256
    signature; `exp` (with leeway); `iss` equals the configured issuer. Then enforces
    the app policy: the email must be present AND Keycloak-verified, and — if
    configured — the authorized party (azp) must match.
    """
    signing_key = self._keys.get_signing_key_from_jwt(token)
    claims = jwt.decode(
        token, signing_key.key,
        algorithms=_ALGORITHMS, issuer=self._issuer, leeway=self._leeway,
        options={"verify_aud": False, "require": ["exp", "iss"]},
    )

• loan-mcp verifies the token against Keycloak’s JWKS and binds the journey to the token’s party — the verified email resolves (via blind index) to a party reference in the Party Directory. With LOAN_MCP_AUTH_ENFORCED=true, a simulate_loan without a verifiable token is refused outright. Fail closed.
• The orchestrator re-verifies the same token independently and confirms the token-derived party matches the customerRef being operated on. This is the defense-in-depth move: a caller who bypasses loan-mcp entirely and hits the REST edge with a valid token for a different party is rejected here too.
• party-api verifies tokens on its one irreversible route — erasure — requiring the identity to be the party itself or a compliance-officer role.

Trade-off callout: verify-when-present vs require. The orchestrator runs ORCH_AUTH_MODE=verify-when-present: a present token is verified and ownership-checked (fail-closed on mismatch), but token-less callers still pass — which keeps the direct-REST e2e suite and internal tools working against the same stack. A deployment without that path should run require. The point is that the mode is an explicit, named decision — not an accident of which middleware happened to be installed.

PII that can actually be forgotten

Loan conversations are PII magnets — and LLM systems multiply the copies: logs, traces, checkpoints, embeddings. This stack’s answer is architectural, not procedural:

Only one service ever holds PII. The party-api — modeled on the BIAN Party Reference Data Directory service domain — stores identity data; every other service holds an opaque partyReference and copies nothing. Names never enter the orchestrator’s database, the agent’s checkpoints, or the loan records.

And that one service can’t read its own database. PII is stored as ciphertext under a per-party OpenBao Transit key; lookups run on blind indexes (HMACs), not plaintext. Right-to-erasure is then a two-step:

1. Destroy the party’s Transit key.
2. Delete the rows.

Step 1 is the interesting one: destroying the key makes every copy of that ciphertext — live rows, WAL, backups — undecryptable at once. That’s crypto-shredding: erasure that doesn’t require rewriting audit history or hunting down replicas. No other service needs to participate, because no other service ever had the data.

On the agent side, the RAG that learns from customer conversations scrubs PII before anything is embedded or stored — the vector store learns problems and solutions, not people.

When the customer gets stuck: layered help, self-improving RAG

A deterministic slot-filling flow has a classic failure mode: the customer answers “well, it depends…” and the flow re-asks the same question forever. This system layers three escapes:

1. Deterministic guidance first — per-slot help text, constraints, examples. Zero LLM cost, always available.
2. A sandboxed LLM helper — a constrained chain that answers the customer’s actual question about the slot, without being allowed to drive the flow.
3. A self-improving RAG — resolved question→solution pairs (PII-scrubbed) are embedded into pgvector; the next stuck customer with a similar question gets the learned answer. A second, curated knowledge_docs collection answers conceptual questions (“how does French amortization work?”) from bundled explainer documents.

Everything fails soft: if pgvector is unavailable, guidance degrades to the deterministic layer — a help subsystem outage never takes down the loan flow.

One production bug from this feature is worth retelling, because it’s invisible until you stream. The helper chains run inside the flow node, and the node appends the final message itself — so the helper’s tokens were streaming to the UI and the final message arrived afterward: every answer appeared twice. The fix is a tagging pattern:

# Tagged as an internal helper: this runs INSIDE loan_flow_node, which appends the
# final message itself, so the streaming layer drops this chain's token stream.
return (prompt | model).with_config(tags=["loan_flow_internal"])

…and the streaming layer filters spans carrying the tag. If you compose LLM calls inside LangGraph nodes and stream with stream_mode="messages", you will hit this.

Running it: fully offline, deterministically

The entire stack boots with one command and zero cloud dependencies:

docker compose -f loan-acquisition-infra/compose/docker-compose.yml up --build
# → https://localhost:8000  (accept the demo CA once)

The deterministic fake LLM is the unsung hero: it drives the same create → offer → simulate → disburse journey every time, which is what makes 77 Cucumber end-to-end scenarios — including durability chaos with Toxiproxy latency/faults and WireMock — reliable enough to run on every change. The agent itself is AgentCore-compatible (/invocations contract), so the same code targets AWS when you want the cloud.

For day-two visibility, pgweb gives you a free window into the DBOS workflow tables and the outbox — you can literally watch a retry schedule execute and an outbox row flip to published.

Conclusion

We went from “an LLM with tools” to a system where the agent is treated as what it really is: a persuasive but untrusted client, wrapped in the same discipline banks have always applied — a deterministic state machine as the single authority, idempotency underneath every retry, identity verified at every hop instead of asserted once, PII isolated and crypto-shreddable, and an event backstop for the failures the happy path can’t see.

The core takeaway is simple: you don’t make an agent safe by prompting it better — you make it safe by building a system that stays correct when the agent is wrong. The LLM proposes; the machine disposes. Everything else in this architecture is that one sentence, applied layer by layer.