Hindsight Gave Our Agent Memory. Guardrails Decide When It Listens. v

The most important line in our accounts-payable agent is the one that ignores memory.

Ledgerwise resolves invoice exceptions — price variances, short shipments, wrong GST rates, surprise surcharges — by recalling how the same vendor's same exception was handled before. That recall is what takes it from 0% to 75% of exceptions handled automatically within three weeks of replayed invoices. But when I worked on the decision policy, almost all of my time went into the opposite question: when should an agent with a good memory refuse to use it?

In finance, a confident wrong answer costs money that doesn't come back. So before writing any more recall logic, we wrote down what memory is never allowed to decide.

The system in one paragraph

A deterministic 3-way match compares the purchase order, the goods receipt and the invoice, and flags exceptions with a measured size: a percentage for price, rupees for a surcharge, units for a short shipment. For each exception the agent asks Hindsight, our memory layer, something like "How were GST mismatch exceptions on invoices from Kaveri Specialty Chemicals resolved, and why?" Every human decision is retained along with the reason the reviewer typed. A policy then decides whether that history is enough to act alone, or whether the invoice goes to a human with the precedents attached.

Recall gives the agent knowledge. The guardrails give it judgement. Here are the five we ended up with, and why each one exists.

Guardrail 1: some decisions are never automatic

NEVER_AUTO = {"BANK_CHANGE", "NO_PO"} if exc.type in NEVER_AUTO: precs = self.recall_precedents(vendor, exc) return ExceptionDecision(exc.to_dict(), "ESCALATE", False, "Guardrail: bank/PO changes always need a human call-back check.", [p.to_dict() for p in precs]) 

A vendor writing "we've moved our banking, please update" is the classic invoice-fraud pattern. In our data, Meridian Electricals has six clean, paid invoices in memory when that letter arrives in week six. A purely similarity-driven agent sees a trustworthy vendor and a familiar invoice. Ours holds it for a call-back to the number already on file.

Notice that it still runs the recall. The history is useful context for the human ("six clean invoices, never changed banks before") — it just doesn't get a vote.

Guardrail 2: the agent never cites itself

Every automatic decision is retained too, tagged decided_by: Ledgerwise agent (auto). That's useful for audit and for Hindsight's reflect to build a vendor brief. But when the policy counts precedent, it filters those out:

precs = [p for p in self.recall_precedents(vendor, exc) if p.is_human] 

Without this line, the failure mode is quiet and compounding. One wrong auto-approval becomes a precedent, the next similar invoice now has more "support", and by the end of the week the agent is confidently agreeing with itself. Memory is only as trustworthy as who wrote it. We have a test that puts three agent approvals in memory and checks that the agent still escalates.

Guardrail 3: no extrapolation beyond what a human approved

Similarity search can't tell 2.8% from 4.2%. Both are "price variance, same vendor, approved". So every memory carries its number, and the policy compares numbers:

approved = [p.magnitude for p in precs if p.decision == "APPROVE"] rejected = [p.magnitude for p in precs if p.decision == "REJECT"] cap = max(approved) * MAGNITUDE_HEADROOM if approved else None # +10% floor = min(rejected) if rejected else None 

Inside the range humans approved, the agent acts. Outside it, it escalates — but it doesn't hand over a blank page. It says which way the history leans:

Before guardrails (what pure recall would do): Sharma Logistics sends a ₹6,800 fuel surcharge. Every earlier surcharge from them was approved. Approve.

After: every approved surcharge was ₹2,400 or less, so ₹6,800 is escalated with "far from anything approved — leaning reject." The reviewer rejects, and that rejection becomes the ceiling for next time.

The same rule makes the agent escalate Tata Steel at +3.1% when humans have only ever approved up to +2.8% — with "leaning approve". The reviewer approves in one click and the learned range widens. Escalating that one cost a few seconds of human time. Extrapolating the other would have cost real money.

Guardrail 4: disagreement resets trust

For yes/no exceptions like a short shipment or a duplicate invoice, the agent only acts when the last three human decisions agree:

recent = precs[-3:] outcomes = {p.decision for p in recent} if len(outcomes) == 1: outcome = outcomes.pop() return ExceptionDecision(exc.to_dict(), outcome, True, ...) return ExceptionDecision(exc.to_dict(), "ESCALATE", False, f"Humans disagreed recently ({', '.join(sorted(outcomes))}).", cited) 

If a reviewer overrides the usual call — maybe the contract changed, maybe the first decisions were wrong — the agent stops acting alone until humans are consistent again. One override shouldn't be ignored, and it shouldn't be blindly followed either.

Guardrail 5: the language model never decides

Ledgerwise uses an LLM (Groq, openai/gpt-oss-120b) to write the explanation a reviewer reads and the email to the vendor. It never makes the pay or don't-pay call. If the model is slow, down, or returns something odd, a plain template takes over and the invoice still moves. Keeping the model out of the decision path made the whole system easier to test: every decision can be reproduced from the precedents alone.

Where Hindsight fits in this

The guardrails only work because the memory layer is precise. A few choices that mattered:

• A skeptical bank disposition. We create the bank with skepticism: 5, literalism: 4, so one approval of a 3% variance isn't generalised into "price variances are fine".
• Recall with source chunks. include_chunks=True returns the exact precedent line behind each extracted fact, so the policy compares real numbers rather than paraphrases.
• Reasons, not just outcomes. Every retained decision carries the reviewer's sentence ("HSN 2839 is contracted at 12% — never pay the 18% version"). That's what escalations show and what Hindsight's reflect turns into a vendor brief.

If memory for agents is new to you, Vectorize's explainer on what agent memory is is a good primer on why this is different from bolting a vector search onto a prompt.

Results

On eight weeks of replayed invoices from six vendors:

The weeks where the rate dips (60% in weeks five and six) are the guardrails working: the +3.1% variance, the ₹6,800 surcharge and the bank change all went to a human.

What I'd tell anyone giving an agent memory

1. Write the guardrails before the recall code. Deciding upfront what memory can never override made every later design choice simpler.
2. Separate who said it from what was said. Retain the agent's own decisions for audit; never count them as evidence.
3. Store magnitudes. "Same vendor, same exception" is not "same decision" if the amount is three times bigger.
4. Escalate with a lean, not a shrug. An escalation that says "leaning approve, closest precedent is X" takes a human seconds, not minutes.
5. Measure wrong automatic calls, not just automation rate. A 56% automation rate with zero wrong calls is worth more than 90% with a few.

The honest limitation: these guardrails are rules we chose for this domain. A different business would need its own list. The pattern carries over even if the rules don't.

Code: https://github.com/SomeswararaoTellakula/ledgerwise
· Live demo: https://ledgerwise-1ydb.onrender.com/

· Shout-out to Code.in and the Hindsight team at Vectorize.v