Why our incident agent remembers bad fixes using Hindsight

Why Our Incident Agent Remembers Bad Fixes
Engineering "negative memory" into on-call triage with Hindsight
Nov 7, 2026
TL;DR: Most runbooks, and most LLMs, tell you what to do. Ours also tells you what not to do, because it remembers which fixes made past outages worse. Here is how we built it with Hindsight agent memory.
At 2:14 AM on a Tuesday, an alert fires for our checkout API. The generic runbook, and every standard LLM prompt, tells the on-call engineer to restart the failing pods and scale the deployment to twenty replicas.
If they follow that advice, they flood the primary database with four hundred new connections, push connection pools into complete starvation, and turn a four-minute degraded service into a forty-minute hard outage.
We built an on-call copilot that does the opposite. It remembers the exact remediation steps that previously failed, cites the historical post-mortems where those failures occurred, and warns the engineer away from harmful actions before giving the proven fix. To do this, we integrated Hindsight into our production incident triage loop.
Below: how the architecture works, how we engineered negative memory into autonomous triage, and what we learned connecting real-time telemetry to institutional history.
How the System Fits Together
Our on-call stack has three layers:

1. Telemetry and synthetic probing. Open-source monitoring via OneUptime continuously runs synthetic health checks against microservice endpoints (such as /health on our payment and checkout services). When error budgets burn or latency spikes, OneUptime packages the alert and fires a webhook.
2. Institutional memory via Hindsight. Rather than treating every incident as a blank slate, or stuffing hundreds of legacy Markdown post-mortems into a stateless context window, we persist verified incidents, runbook resolutions, and team policies in a dedicated Hindsight memory bank.

3. A grounded triage agent. When an alert fires, the agent runs semantic recall queries against Hindsight. If it finds historical precedent, it synthesizes a five-minute tactical briefing:
   ◦ Likely root cause, with a confidence score
   ◦ Matching past incidents, citing exact incident IDs
   ◦ Do first, with copy-pasteable, verified shell commands
   ◦ Do NOT do, listing explicit pitfalls and negative-memory warnings
   ◦ Escalate, naming who to page based on organizational policy
   When the outage resolves, the engineer closes the loop. The validated resolution, along with what failed during troubleshooting, is retained back into Hindsight with UTC timestamps, so every subsequent shift benefits.
   The data flow at a glance
   OneUptime synthetic probes (HTTP 503 / latency spikes)
   |
   v
   Webhook intake (:8787, normalized incident stream)
   |
   v
   Hindsight memory core
   retain() - store incidents, policies, post-mortems
   recall() - semantic search over past outcomes
   reflect() - cross-history synthesis and patterns
   |
   v
   Incident copilot briefing

   • Baseline vs. grounded comparison
   • Cited incident IDs, pitfalls caught, time saved
   • "Avoid this" negative-memory warnings
   • Exact rollback commands (human in the loop) The Core Idea: Engineering Negative Memory Most agent-memory implementations suffer from positive confirmation bias: they retrieve what worked. In distributed-systems SRE, knowing what worked is only half the battle. Knowing what made things worse is what prevents the really expensive outages. When engineers write post-mortems, the most valuable section is rarely the final fix. It is the log of dead ends, harmful assumptions, and destructive remediation attempts. Take an outage caused by database connection pool exhaustion: pod restarts temporarily mask the failure while threads queue up, and scaling out replicas multiplies total database connections until PostgreSQL hits max_connections. To capture this, we built our memory schema around explicit negative memory. When an incident closes, we isolate a failed field and write it into its own labeled section: ` def outcome_document( service: str, title: str, root_cause: str, fixed: str, failed: str, ttr_minutes: int, resolver: str, severity: str = "Sev 2", ) -> tuple[str, str]: now = datetime.now(timezone.utc) incident_id = f"INC-{now:%y%m%d%H%M%S}" failed_items = [part.strip() for part in failed.replace("\n", ";").split(";") if part.strip()] failed_block = "\n".join(f"- {item}" for item in failed_items) if failed_items else "- None recorded." fixed_one_line = " ".join(fixed.split())

   document = (
   f"Incident ID: {incident_id}\n"
   f"Date: {now.isoformat().replace('+00:00', 'Z')}\n"
   f"Service: {service}\n"
   f"Severity: {severity}\n"
   f"Title: {title}\n"
   f"Symptoms: Reported through operational intake by {resolver}.\n"
   f"Root cause: {root_cause}\n"
   f"Resolution:\n1. {fixed_one_line}\n"
   f"What failed (made it worse):\n{failed_block}\n"
   f"TTR: {ttr_minutes} minutes\n"
   f"Responder: {resolver}"
   )
   return incident_id, document
   `

By isolating What failed (made it worse) inside the structured payload, Hindsight indexes not just the vectors for symptoms and fixes, but the semantic space around harmful interventions too.
Implementation

1. Retain and recall with Hindsight
   We wrapped the client following the official Hindsight docs. Bank directives give the memory store a site-reliability persona, and idempotent document IDs mean that re-seeding or post-incident updates never create duplicate records.
   `
   class MemoryStore:
   def init(self, base_url: str = "http://localhost:8888", bank_id: str = "incident-response-agent"):
   self.base_url = base_url
   self.bank_id = bank_id
   self._client = None

   def retain(
   self,
   content: str,
   context: str = "",
   timestamp: str | None = None,
   document_id: str | None = None,
   update_mode: str = "replace",
   ) -> None:
   if not self._client:
   raise RuntimeError("Memory backend is not connected.")

   self._client.retain( bank_id=self.bank_id, content=content, context=context, timestamp=timestamp or datetime.now(timezone.utc).isoformat(), document_id=document_id, update_mode=update_mode, ) 

   def recall(self, query: str, limit: int = 5) -> list[Memory]:
   if not self._client:
   raise RuntimeError("Memory backend is not connected.")

   results = self._client.recall(bank_id=self.bank_id, query=query, limit=limit) return [Memory(text=item.text, score=getattr(item, "score", 0.0)) for item in results] 

   `

2. Strict behavioral guardrails in the prompt
   LLMs love to hallucinate plausible-sounding Kubernetes commands (kubectl scale --replicas=50). In a high-pressure incident, an engineer might run a hallucinated command without double-checking.
   So our prompt enforces a strict contract: commands may only be suggested if they appear verbatim in retrieved memory. If memory has no verified command, the agent describes the check in words and omits the code block.
   You are Incident Memory Agent, an on-call incident-response copilot.

Your job: turn a production alert plus retrieved memories of past incidents
and team policies into a briefing that helps an on-call engineer act in the next 5 minutes.

Rules:

• If RETRIEVED MEMORY is empty: output a generic runbook and plainly state no incident history was available. Do not cite any incident ID.
• Never invent an incident ID, command, policy, responder, or metric. Use only the alert and retrieved memory.
• Every command in Do first must appear verbatim in RETRIEVED MEMORY.
• Never recommend an action that RETRIEVED MEMORY says made things worse.
• Output section Do NOT do listing actions that made past incidents worse, each citing its incident ID and consequence.
• Fail loud, never fake memory Early on, we made a subtle design decision: no in-process mock fallbacks. If the Hindsight service is unavailable, the copilot refuses to run memory-grounded queries and tells the operator that memory is offline. Faking memory with ungrounded keyword matching gives on-call engineers false confidence in the middle of an emergency. If institutional memory is down, the agent should say so, loudly. A Real Incident Walkthrough To validate the architecture, here is a scenario on our checkout-api service. The injected outage A new release (v3.4.1) rolls out to production. The developers cut the HikariCP connection pool from 40 to 12 and added a synchronous foreign-exchange lookup inside an active database transaction. Under normal load, latency degrades. Then connection-pool timeouts cascade: 2026-11-06T18:41:02Z checkout-api HikariPool-1 - Connection is not available, request timed out after 30000ms 2026-11-06T18:41:15Z deploy-bot INFO v3.4.1 rolled out 6m before alert OneUptime's synthetic probes detect HTTP 503 errors and open incident INC-OU9001. Without memory: the generic LLM baseline Given the alert with no retrieved memory, a standard completion returns runbook boilerplate: • Likely root cause: High incoming request volume causing thread pool starvation. • Do first:
  1. Scale deployment replicas from 4 to 20: kubectl scale deployment checkout-api --replicas=20
  2. Rolling restart of all checkout pods: kubectl rollout restart deployment checkout-api • Do NOT do: No documented harmful actions. With Hindsight: the grounded agent With recall enabled, the agent surfaces two historical incidents: INC-2139 (a pool-exhaustion event during Black Friday) and INC-2041 (an FX latency cascade). The briefing changes completely: • Likely root cause: Hikari connection pool exhaustion caused by blocking FX calls introduced in deploy v3.4.1 (confidence: high). • Matching past incidents: ◦ INC-2139: Pool exhaustion after the v3.2.0 deploy caused identical 30-second timeouts. ◦ INC-2041: Latency spike in payment processing due to a synchronous downstream call. • Do first:
  3. Roll back the deployment immediately: deploy rollback --service checkout-api --to v3.4.0 --reason "pool exhaustion after v3.4.1"
  4. Verify active database connections:

• ⛔ Do NOT do (negative memory):
◦ Do not scale replicas up. In INC-2139, scaling checkout-api to 20 replicas consumed all 480 connections on the primary PostgreSQL instance and crashed the orders and auth services.
◦ Do not restart pods blindly. In INC-2139, restarting pods re-exhausted the connection pools within 180 seconds and dropped all in-flight shopping carts.
• Escalate: Page Marcus Bell (Checkout Team Lead) per policy POL-104.
The difference is night and day. The generic agent offered textbook advice that would have turned a minor degradation into a site-wide disaster. The memory-grounded agent prevented it and supplied the exact rollback command.
Cross-Incident Reasoning with reflect()
Beyond point-in-time response, we wanted to surface systemic risks across dozens of outages. Vector search (recall) struggles with meta-questions like "Which remediation actions have repeatedly made things worse across all services?"
For that we use Hindsight's reflect() API, which reasons across the whole document corpus instead of returning only the top-k snippets:
answer = store.reflect("Which remediation actions have repeatedly made things worse across our infrastructure?")
Hindsight returns a structured synthesis backed by historical evidence:

1. Unchecked pod restarts during connection pool starvation (INC-2139, INC-2041). Restarts clear the symptoms for less than three minutes while thrashing database connection handshakes.
2. Flushing Redis caches during high auth latency (INC-2204). Issuing FLUSHALL during an authentication spike stripped the cache layer and sent thousands of uncached token verifications straight to PostgreSQL, triggering a total auth outage.
3. Database restarts during peak write spikes (INC-1988). Forcing an immediate restart of the primary database triggered a 22-minute Write-Ahead Log (WAL) replay during which write transactions were completely locked. This turns post-mortems from dead Google Docs into continuous architectural intelligence. Four Lessons from Building with Agent Memory
4. Grounding requires verbatim constraints If you let an LLM "adapt" shell commands found in memory, it will introduce subtle flag incompatibilities or invent flags that don't exist. Require that executable commands appear verbatim in the retrieved memory payload, and flag any command needing human approval with a clear UI badge.
5. Time-aware context prevents stale fixes A resolution that worked three years ago on Kubernetes 1.18 may break a cluster running 1.30. Passing explicit ISO timestamps into Hindsight's timestamp parameter lets newer post-mortems carry appropriate weight when retrieving context for modern deployments.
6. Make closing the loop frictionless If adding post-mortem memory means writing a five-page wiki, engineers will skip it. We tie retention directly to the incident-resolution interface: fields are extracted from the OneUptime webhook, and a 30-second closure form captures the rest. New knowledge becomes a natural byproduct of resolving the ticket.
7. Separate memory banks by domain Don't dump API docs, customer support tickets, and production post-mortems into one global bank. Isolating incident response in its own bank (incident-response-agent) kept semantic density high and stopped irrelevant documentation from polluting high-urgency briefings. What's Next Reliable AI agents in production cannot be amnesiac. By pairing real-time synthetic monitoring with structured agent memory, we stopped repeating the same operational mistakes every on-call rotation. If you're building autonomous SRE agents, or want to give LLMs persistent, structured recall, check out the Hindsight GitHub repository and see how Vectorize agent memory changes the way agents learn from real-world outcomes.