I Built an Open-Source Tool to Test What Happens When MCP Servers Fail

I've been spending a lot of time exploring MCP servers and AI agents lately.

One thing that caught my attention: we spend a lot of time testing whether tools work, but what about when they don't?

What happens when an MCP tool crashes? When it receives invalid arguments? When a protected endpoint receives an unauthorized request?

And more importantly, does the server actually report those failures correctly?

That's what led me to build MCP Verdict.

The problem

Imagine an MCP tool fails internally but returns something like this:

{ "isError": false, "content": [ { "type": "text", "text": "Operation failed" } ] } 

The message says the operation failed, but the MCP response reports success.

That inconsistency can cause problems for clients and agents relying on the tool's result.

And it's exactly the kind of behavior that's worth testing before deploying an MCP server.

Introducing MCP Verdict

MCP Verdict is an open-source Python CLI that lets developers define expected MCP server behavior and check what actually happens.

Instead of writing separate scripts for every failure scenario, you describe your expectations in YAML.

For example:

server: command: - python - examples/basic_stdio/server.py tool: name: ping arguments: {} expected: success 

Then run:

mcp-verdict check --config verdict.yaml 

MCP Verdict connects to the server, executes the tool, and checks whether the actual result matches your expectation.

What it supports

The first release includes:

• STDIO and Streamable HTTP — Test real MCP servers using the official Python SDK.
• Tool outcome checks — Verify success and expected tool errors using MCP's isError field.
• HTTP security checks — Test selected authentication and authorization responses, including HTTP 401 and 403.
• YAML configuration — Define repeatable checks with configurable timeouts.
• JSON reporting — Generate structured results for automated workflows.
• CI-friendly exit codes — Distinguish PASS, FAIL, and execution ERROR.

Testing incorrect error reporting

Here's another example.

Suppose a tool is expected to return an MCP tool error, but instead reports success.

MCP Verdict can detect that mismatch:

known-failure: FAIL: Expected tool_error, observed success Summary: 0 passed, 1 failed 

This is the behavior I find most interesting: testing not just whether an MCP tool works, but whether it fails correctly.

Try it

MCP Verdict v0.1.0 is now available on PyPI.

Requires Python 3.11 or 3.12.

pip install mcp-verdict 

Verify the installation:

mcp-verdict --help 

For runnable examples, clone the repository:

git clone https://github.com/kuruash/mcp-verdict.git cd mcp-verdict mcp-verdict check --config examples/basic_stdio/verdict.yaml 

The repository also includes calculator, intentional error-handling, and local HTTP security examples.

What I learned

Building this made me appreciate how much work goes into handling the unhappy paths of software.

Connecting to a server is one thing. Handling timeouts, unexpected disconnects, malformed responses, authentication failures, and cleanup reliably is a different challenge.

I also learned a lot about Python packaging, GitHub Actions, testing with real subprocesses, and publishing through PyPI Trusted Publishing.

There's still plenty to improve. MCP Verdict isn't a comprehensive MCP security scanner, and it doesn't attempt to audit arbitrary OAuth/JWT implementations.

For now, the goal is simple: make failure scenarios reproducible and help developers catch regressions.

I'd love some feedback

If you're building or maintaining MCP servers, I'd love to know:

What's the most frustrating failure scenario you've encountered while testing an MCP server?

I'm especially interested in edge cases that MCP Verdict doesn't handle yet.

GitHub: https://github.com/kuruash/mcp-verdict

PyPI: https://pypi.org/project/mcp-verdict/

Issues, feedback, and contributions are welcome!