PII That Can Actually Be Forgotten — Blind Indexes and Crypto-Shredding in an AI Agent Stack

A DELETE doesn’t delete. Postgres keeps your customer’s name in heap pages until VACUUM, in every WAL segment, in every backup, in every replica. Here’s an architecture where “right to erasure” is a one-key operation — and where the AI layer never gets PII to leak in the first place.

LLM systems are PII multipliers. A name typed into a chat can end up in graph state, in a checkpointer table, in logs, in traces, in a vector store — and then the customer invokes their right to erasure, and you discover that “delete the user” now means a forensic sweep across seven data stores and a backup archive you can’t rewrite.

This stack takes the opposite bet, made of two architectural decisions:

1. Only one service ever holds PII. Everything else carries an opaque reference. 2. That one service stores nothing it can read without a key someone else custodies — so erasure is the destruction of one key.

The service is party-api, modeled on the BIAN Party Reference Data Directory service domain. The key custodian is OpenBao Transit (“encryption as a service” — key material never leaves it). And the AI layer gets its own, independent control: scrub before anything is stored or embedded.

One vault everyone else references by an opaque id; one key-keeper the vault cannot live without; and erasure reduced to destroying a single key — which turns every copy, in every backup, unreadable at the same instant.

Why a row DELETE can’t prove erasure

The schema’s header comment states the threat model better than most compliance decks:

// Row DELETE alone cannot PROVE erasure: the plaintext lingers in Postgres pages until VACUUM, and
// in every WAL segment and base backup until those age out. So no PII column stores plaintext.
// Each PII-bearing row stores:
//
//   * `*_ciphertext`  — the row's PII as one JSON blob, encrypted under THAT PARTY's key held in
//                       OpenBao Transit. Erase destroys the key, which makes every copy of the
//                       ciphertext (live, WAL, backup, replica) undecryptable at once. Proving
//                       erasure collapses to proving one key-deletion event.
//   * `*_bidx`        — a blind index: keyed HMAC of the normalized searchable value, under a
//                       SERVICE-WIDE key that never leaves OpenBao.

That’s the whole design in two bullet points: per-party keys make erasure granular; keyed hashes make search possible without plaintext. The database contains ciphertext and one-way HMACs. A dump of party-api’s memory, environment, or database yields no key — Transit encrypts and decrypts on its side of the wire.

The three flows that make it work — register, lookup, erase — in full detail:

Blind indexes: equality search over data you refuse to store

“Find the party for alice@example.com” has to work — it’s how a verified login binds to a bank party, and how an auditor resolves a human to a reference. The blind index is a keyed HMAC-SHA256 of the normalized value (trim().toLowerCase(), applied identically at write and at search), stored in a *_bidx column with a covering B-tree index. Search hashes the query term via Transit and does an equality match.

Two deliberate choices hide in that design:

• The HMAC key is service-wide, not per-party — the code says why: “a per-party key would make ‘find this email across all parties’ impossible.” You don’t know which party you’re looking for; that’s the point of the lookup.
• It’s an HMAC, not deterministic encryption — one-way. Even if the column leaks, it reveals nothing and reverses to nothing.

And the residual risk is written down instead of wished away:

// Service-wide HMAC key for blind indexes. NEVER destroyed by erasure: erasing a party deletes its
// blind-index ROWS instead. Residual risk, accepted and documented: a blind-index hash surviving in
// an aged DB backup is pseudonymous (keyed hash, key held only by OpenBao) but not erased until
// that backup expires — backup retention is the governing control there.

Erasure: key first, rows second — because of how each half fails

The erasure endpoint is the one irreversible route in party-api, and it executes in a precise order:

async erase(partyReference) {
  await requireLive(deps.db, partyReference);
  await deps.cipher.destroyKey(partyReference);        // 1. crypto-shred
  const erasedAt = deps.now();
  return deps.db.transaction(async (trx) => {          // 2. tombstone
    // Whole rows, not nulled columns — a column added later cannot be forgotten by a DELETE.
    for (const table of PII_TABLES) {
      await trx(table).where({ party_reference: partyReference }).del();
    }
    await trx(TABLES.ENTRY).where({ party_reference: partyReference }).update({
      status: 'Erased', erased_at: erasedAt, /* … */ person_details_ciphertext: null,
    });
  });
}

The ordering is a failure-mode argument, not a style choice: destroy the key first, and a crash before the row-delete leaves a state that a retry completes. Delete rows first, and a crash before key destruction could strand a permanently half-erased party — whose retry the requireLive guard then refuses. When an operation is irreversible, you design it around where it can be interrupted.

What remains is a tombstone: the entry survives with status: 'Erased', an erasedAt date, and no identity whatsoever. That’s deliberate — every other service still holds the partyReference, and a dangling reference must be explainable (“erased on 2026-07-17”), not indistinguishable from data loss. Audit history everywhere keeps every structural fact — party, amount, state, time — and permanently loses the ability to say who.

Downstream, the orchestrator’s party gate closes the loop: a new journey for an erased party is refused with a 422, fail-closed, while historical journeys keep pointing at the reference by design.

Who may push the button

Erasure is authenticated with the same rigor as the data it destroys. The route demands a bearer token, verified against Keycloak (RS256, pinned issuer, email_verified, authorized-party check), and then applies a two-ground rule:

if (principal.realmRoles.includes(complianceRole)) {
  return;                                   // ground 1: compliance-officer
}
const own = await config.resolver.referencesForEmail(principal.email);
if (own.includes(targetPartyReference)) {
  return;                                   // ground 2: it's your own party
}
throw new ForbiddenError(
  'This identity may erase only its own party; erasing another party requires a compliance role.',
);

No token → 401. Unverifiable token → 401 (the reason stays in the logs; the wire message is deliberately coarse). Valid token, wrong person → 403. The e2e suite curls all three outcomes and then asserts the tombstone directly in the database: entry present, status Erased, zero PII rows.

Note the elegant self-reference: ground 2 uses the blind index itself — the verified email is hashed and matched to prove “this is your own party.” The privacy mechanism doubles as the authorization mechanism.

The AI plane: don’t shred what you never stored

Crypto-shredding protects the system of record. The conversational plane — LLM prompts, LangGraph checkpoints, the self-improving RAG — is a different data plane with its own control, applied at a single ingestion seam:

def _redact_and_log(text: str) -> str:
    """
    Scrub PII from inbound USER text at the single ingestion seam — before it becomes
    a message in graph state (which the checkpointer persists) or reaches the LLM.
    Logs the categories redacted, never the values.
    """

The redactor is pure regex + Luhn: SSNs, payment cards (Luhn-validated so a $20,000 loan amount is never mistaken for a card number), keyword-anchored account numbers, passwords and PINs. Before anything reaches the pgvector knowledge base, a second pass generalizes the text — every numeric collapses to <num> — so the KB stores problems and solutions, never people or amounts. And Bedrock Guardrails add a model-boundary backstop, anonymizing emails and phones and blocking cards and SSNs in both directions.

The two planes compose into a tidy claim: the agent never sends raw identifiers to a model or a vector store, and the system of record never stores plaintext identity at all. Erasure therefore touches exactly one service, and proving it means producing one key-deletion event.

Conclusion

Most “GDPR-ready” architectures are a promise to go looking for data when asked. This one is a refusal to spread it: one PII service, opaque references everywhere else, ciphertext under per-party keys, search via one-way hashes, erasure as key destruction plus a tombstone, and an AI layer that scrubs before it stores.

The takeaway generalizes past banking: decide where PII is allowed to exist before you build, make that place unable to read its own database without external keys — and then the right to be forgotten stops being an archaeology project and becomes an API call.