How to Audit a Multi-Agent CrewAI Crew: Find the Handoff That Broke the Output

AI agents are moving from demos into real work, and a single agent that gives a wrong answer is a problem. A crew that gives a wrong answer is a mystery. The researcher passed notes to the analyst, the analyst passed a draft to the writer, and the customer received a paragraph that nobody can fully explain. Each step looked reasonable on its own. The failure lives in the handoffs.

This guide shows how to make a CrewAI crew auditable, so that agent debugging, an agents audit, and AI compliance questions all draw on one record. We use ZizkaDB, an open source causal DB for AI agents, with its CrewAI logger, its MCP server, and two GitHub Actions. The same approach suits LangGraph and other frameworks. The code is short on purpose, and the focus is on what each step buys you in business terms.

A note on scope: details such as how delegation between agents is represented are best confirmed in the reference example at examples/crewai-agent/, so check the current documentation before you rely on specifics.

Why crews fail differently from single agents

Multi-agent systems add a failure class that single agents lack: context corruption between roles. A researcher finds the right fact, summarizes it slightly wrong, and hands that on. Every later agent then reasons correctly from a faulty premise. The output looks confident and fluent, and the agent that appears responsible is the last one, rarely the one that caused the damage.

Two patterns show up again and again. The telephone effect is small distortions at each handoff that compound. Authority confusion is a downstream agent treating an upstream guess as a verified fact.

Neither shows up in the final text. They show up in the path, which most teams do not keep.

Auditability is not the same as AI observability

Most teams already have some AI observability: traces, latency, token counts, and error rates. It answers a useful question, which is whether the system is healthy right now. It helps you notice problems, and it is usually sampled, summarized, and short lived.

Agent auditability answers a different question: can you reconstruct what happened afterwards, explain it, and prove the record was not changed? That asks for more than a trace:

A complete audit trail, with every task recorded and not a sample.
Causal links, so each step points to the step that caused it.
Integrity, so the audit log can be checked for edits, insertions, and deletions.

The two are complementary. LLMOps teams own deployment, evaluation, and monitoring, and auditability adds the evidence layer underneath.

Where the EU AI Act comes in

The EU AI Act makes this concrete for high-risk systems. Article 12 says such systems must technically allow for the automatic recording of events over the lifetime of the system, and the logs serve purposes that include identifying risk situations, supporting post-market monitoring, and monitoring the operation of the system. Application dates for the high-risk obligations have been shifting, and one tracker lists 2 December 2027 for systems under Annex III, so check the current timeline, and whether your use case counts as high risk, with your legal counsel.

Outside the EU AI Act, enterprise buyers and auditors ask for the same evidence, which is why agent auditability is becoming a sales requirement as well as a legal one.

What we will build

By the end you will have a recorded CrewAI crew, inspectable handoffs and memory, a debugging routine, verified tamper-evident logs, a pull request workflow, editor access, and an evidence pack.

Step 1: Start the causal DB

You need Docker. This starts a local ZizkaDB instance, and the first image pull can take a few minutes:

bash
curl -fsSL https://raw.githubusercontent.com/Zizka-ai/ZizkaDB/main/scripts/quickstart-remote.sh | bash

On localhost there is no signup and no API key. The API runs at http://localhost:8000 and the dashboard at http://localhost:3001.

One constraint matters for crews, so it is worth stating now. The open source edition records one agent per install. Recording each role as its own agent, fleets, and agent-to-agent recording are ZizkaDB Cloud features. On the open source edition, log the whole crew under one agent name and carry the role as a field on each event, which answers almost every debugging question.

Step 2: Record the crew run

Install the SDK and the CrewAI logger. Both are on PyPI:

bash
pip install zizkadb-sdk zizkadb-crewai crewai

The logger records the crew lifecycle with three methods: log_kickoff for the start of the run, log_task for each task, and log_output for the final result. Events chain automatically through the logger's last_event_id, so you can leave out the parent link and each event attaches to the previous one. Here is the pattern from the package documentation, with the task loop written out:

python
from zizkadb import ZizkaDB
from zizkadb_crewai import ZizkaDBCrewLogger

async with ZizkaDB(host="http://localhost:8000") as db:
logger = ZizkaDBCrewLogger(db=db, agent="research-crew")

await logger.log_kickoff(goal="Prepare a supplier risk summary") result = await crew.kickoff_async() for task_output in result.tasks_output: await logger.log_task( description=task_output.description, output=str(task_output.raw)[:2000], ) await logger.log_output(str(result)) (await db.why(logger.last_event_id)).print() 

The last line is the payoff. It walks back from the final output through every task to the kickoff, so a run becomes a causal tree and not a pile of unrelated lines.

Two honest details. First, this example records tasks after the crew finishes, so the timestamps reflect logging time; check the repository example for logging inside task callbacks. Second, the logger accepts extra keyword fields, so add the role and model name, using the attribute names from your CrewAI version.

When a customer asks why the summary said what it said, you read a record instead of reconstructing from memory.

Step 3: Make every handoff carry its evidence

The cheapest improvement to a crew is to treat each handoff as a recorded object. For each task, log four things beside the output: which role produced it, which sources it used, how it described its confidence, and whether it was asked to do anything it skipped.

python
await logger.log_task(
description=task_output.description,
output=str(task_output.raw)[:2000],
role=str(task_output.agent),
sources=["supplier_db", "news_search"],
stated_confidence="medium",
)

This protects against authority confusion, because the record shows whether the researcher ever claimed a fact firmly. If most wrong answers trace back to tasks with medium confidence and no sources, you have a rule to enforce: do not let the next role proceed on a thin handoff.

Truncate large outputs and redact sensitive fields before storage, because agents quote customer details freely.

Step 4: Give the crew agent memory you can audit

Crews often share a memory layer so that later runs benefit from earlier ones. That is useful, and it is also a risk: a wrong conclusion stored once can mislead every crew that follows. Agent memory that nobody can inspect turns one bad run into a standing problem.

Treat LLM memory as part of the record, because LLM memory is a store of facts and often of personal data. The SDK has two ways to read it. The search function returns events matching a query, and the context function returns a prompt-ready block of relevant memory for a task:

python
hits = await db.search("supplier Acme payment terms", agent="research-crew")
block = await db.context_for("research-crew", task="Summarize risks for supplier Acme")

A few documented behaviors make shared memory safer:

Replaced facts leave retrieval. Writing an event with a supersedes reference to an earlier event keeps the old fact in the log but removes it from context and search, so a corrected fact replaces the old one.
Importance and recency rank results, with recency halving every 30 days by default.
Search works without a hosted embedding service, using keyword search by default and local embeddings if you want semantic modes.
python
await db.log(
agent="research-crew",
event="memory_written",
data={"text": "Acme moved to net-60 terms", "subject_id": "supplier_113"},
supersedes=old_event_id,
importance=0.8,
)

Then close the loop on provenance. Log an event listing which memories were supplied to the crew at kickoff, and make the kickoff a child of it. I could not confirm that the context function records this automatically, so check one session in the dashboard and add the event yourself if needed. With that link in place, causal lineage can show which stored note fed a wrong answer, which is the question you will ask most often.

Step 5: Debug with causal lineage and session replay

With the run recorded, agent debugging follows a routine. Use two tools together.

Session replay shows one run from first event to last, in order. In the dashboard, open Activity, choose the crew agent, and open the session. Read it like a transcript. Find the first task whose output differs from what you expected, since the visible error is often two handoffs downstream of the cause.

Causal lineage explains a specific step. Run it from the terminal or in code:

bash
zizkadb why
python
(await db.why(event_id)).print()

You get the chain of inputs, context, and prior outputs behind that step. Classify the cause as one of four types: bad input, bad context, a failing tool, or a reasoning error. Each has a different fix, and naming the type builds a record of which failures recur.

For the opposite question, what did this step cause, the API has an impact route, GET /v1/events/{event_id}/impact, which walks the graph downstream. It is valuable for scoping. If one weak research task fed three later reports, you want to know before you reply to the customer, and before the other reports are used.

A worked example: a supplier summary claimed a wrong payment term. Replay shows the researcher had it right and the analyst had it wrong. Lineage for the analyst task points to a memory note from three weeks earlier that was never marked superseded. The cause was stale memory, not a reasoning error, and the fix was a supersedes write. Without the record, the team would have rewritten the writer prompt, which was never the problem.

Step 6: Use time travel debugging for old crews

Incidents are rarely investigated the day they happen. By the time someone asks about a run from six weeks ago, the role prompts have changed, a tool has been swapped, and the model may have been upgraded. Debugging against today's crew gives you the wrong answers.

Time travel debugging fixes that. The at function reconstructs what the agent knew at a given timestamp:

python
state = await db.at("research-crew", "2026-09-15T09:00:00Z")

It rebuilds state from state events that you log, with the newest write per key winning, so its value depends on what you record. Log each role prompt version, the model name, the tool list, and the task ordering whenever they change. Then the question what did this crew look like that day becomes a query. The exact event format is in the SDK documentation, so check it before you depend on it.

For crews this matters more, because a change to one role prompt can alter every role after it. Combine the tools in an investigation: use session replay to find the step, causal lineage to explain it, and time travel debugging to confirm which configuration was live.

Step 7: Make the audit log tamper-evident, and verify it

An audit log is only worth as much as your ability to show it was not changed. ZizkaDB addresses this in layers, according to its changelog:

Each event stores a SHA-256 checksum computed over its content, agent, parent link, session, timestamp, sequence number, and metadata.
Each new event also stores a chain hash that links it to the agent's previous event. Deleting, inserting, reordering, or editing a row breaks the chain.
A verify call checks the whole chain and every checksum, and returns whether it is intact, a summary, any problems, and a chain head.
python
report = await db.verify_chain("research-crew")
print(report)

Store the chain head somewhere independent of the database. Later you can pass it back, with a sequence number, to check only newer events and show that older history was not rewritten. That is what makes tamper-evident logs useful as evidence and not only as a feature.

Be precise about the claim. Checksums and chains show that records have not changed since they were written. They do not show the content was true, and the external anchor matters because someone with full database access could rewrite everything.

Step 8: Put it in your LLMOps pipeline

Two Marketplace actions bring this into pull requests and scheduled jobs. Both start ZizkaDB inside the runner by default, with no account, and run your test command.

ZizkaDB Why posts the causal chain of failing agent tests to the job summary and the pull request. For a crew, that means a failing test shows which task produced the bad output, in the review itself:

yaml
permissions:
contents: read
pull-requests: write

steps:

• uses: actions/checkout@v4
• uses: actions/setup-python@v5 with: { python-version: "3.12" }
• run: pip install -r requirements.txt zizkadb-sdk zizkadb-crewai
• uses: ZIZKA-AI-SL/zizkadb-why-action@v1 with: run: pytest tests/crew

ZizkaDB Audit verifies each agent's hash chain, exports the history, and writes a report attested by GitHub. Pointed at a running instance on a schedule, it produces recurring evidence:

yaml
on:
schedule: [{ cron: "0 6 * * 1" }]
jobs:
audit:
runs-on: ubuntu-latest
permissions: { contents: read, id-token: write, attestations: write }
steps:
- uses: ZIZKA-AI-SL/zizkadb-audit-action@v1
with:
host: https://zizkadb.example.com
api-key: ${{ secrets.ZIZKADB_API_KEY }}

Anyone can check the report with gh attestation verify report.md -R your-org/your-repo.

Practical points: pin the image version and third-party actions, and treat a run that recorded zero events as a failure. The scheduled run against your live instance, not the pull request run, produces evidence about real history.

If you also run LangGraph agents, the same actions and record work for both, so one evidence process covers your whole estate.

Step 9: Add the MCP server to your editor

The same capabilities are available inside MCP-capable editors and clients. The ZizkaDB MCP server is listed in the official MCP Registry as io.github.ZIZKA-AI-SL/zizkadb-mcp, version 0.1.12 at the time of writing, installed from PyPI as zizkadb-mcp over stdio. To configure it by hand, add this to your client's config, such as .cursor/mcp.json, and pin the version you reviewed:

json
{
"mcpServers": {
"zizkadb": {
"command": "uvx",
"args": ["zizkadb-mcp==0.1.12"],
"env": { "ZIZKADB_HOST": "http://localhost:8000" }
}
}
}

The tools cover logging, memory search, causal lineage, past state, and chain verification. For a crew developer, the practical gain is asking the editor why a task produced its output without leaving the code. An MCP server is code that runs on your machine, so confirm the registry name, pin the version, and review upgrades.

Step 10: Assemble the evidence pack for an agents audit

When an agents audit arrives, or a customer sends an AI compliance questionnaire, you want to answer from a prepared package and not from memory. Map each typical question to evidence you already produce:

Question Evidence
What did the crew do, and when? Audit trail and session replay of selected runs
Why did a role produce that output? Causal lineage report for the task
What was configured at the time? Time travel debugging output and logged state
How do you monitor behavior over time? Baseline and drift reviews, with dates
Can you show the record was not altered? Chain verification and attested audit reports

This pack supports your technical documentation and complements, not replaces, your risk management system and any formal assessment. It shows what happened, and people decide whether it was acceptable. Rehearse quarterly: pick a run from three months ago and ask a colleague outside the build team to answer a question using only the pack.

What it saves

These figures are modeled assumptions, not measured results. Replace them with your own.

Assume a team runs crews that produce 10 wrong or disputed outputs a month, at 100 euro per engineering hour. Without a recorded path, each takes about 5 hours to diagnose, because the cause hides behind several handoffs, which is 5,000 euro a month. With session replay and causal lineage, each takes about 1.5 hours, which is 1,500 euro. That saves about 3,500 euro a month, or 42,000 euro a year.

Audit work adds more: four evidence requests a year at about 4,800 euro each fall to about 1,600 with a standing evidence pack.

Setup is a few days of engineering. Self-hosted ZizkaDB is free under AGPL-3.0, and managed cloud is listed at 29 euro a month for Pro and 69 euro for Team. The larger value is harder to price: shorter sales reviews, fewer drawn-out disputes, and the confidence to change role prompts and models more often.

Common mistakes to avoid

Blaming the last agent. The writer is usually where the error becomes visible, not where it started. Follow the chain back.

Letting thin handoffs pass. If a task returns little and the next task proceeds anyway, you have created a silent skip. Record sources and confidence, and enforce a minimum.

Treating AI observability as auditability. Traces help you notice problems, and they rarely prove anything.

Not logging configuration. Time travel debugging only works for what you recorded.

Storing the chain head only inside the database. Keep it independent.

Expecting tools to judge correctness. They explain and verify, and people decide.

Start this week

Take one crew you already run, ideally one with three or more tasks. Start the local instance, add the logger around your kickoff, and run a request. Open the session in the dashboard, then pick a recent wrong answer and run why on it. If the chain points at the cause in under a minute, add role and source fields to each handoff, log your prompt versions as state, and add the Why action to your next pull request. Schedule the Audit action once you have a live instance.

Want to test ZizkaDB on your agent? Try our open source version here: https://github.com/ZIZKA-AI-SL/ZizkaDB

Interested in a design partnership? Fill in the form on our site or reach me directly at founder@zizka.ai

Sources: zizkadb-crewai on PyPI · ZizkaDB Python SDK wiki · ZizkaDB changelog · ZizkaDB Why on GitHub Marketplace · ZizkaDB Audit on GitHub Marketplace · zizkadb-mcp on PyPI · AI Act Article 12 · ZizkaDB repository