Promptfoo vs DeepEval: The Same Support Bot, Tested Both Ways

If you test LLM apps, you will run into Promptfoo and DeepEval within a week. Both are free and open source, both evaluate LLM output, and the comparison posts make them sound interchangeable. They are not. The quickest way to see the difference is to point both at the same app.

So I wrote a tiny stand-in "support bot" and tested it twice: once with Promptfoo, once with DeepEval. No AI model is involved, so you can run everything below in a couple of minutes for free. (Versions: Promptfoo 0.125.1, DeepEval 4.3.4.)

The app under test

A fake support bot with two known answers and a fallback:

# Stand-in for your real LLM app ANSWERS = { "What is your refund window?": "30 days", "Do you ship to Canada?": "Yes, we ship to Canada in 5-7 business days.", } 

In a real project this would call your model or your RAG pipeline. The testing side stays the same.

Version 1: Promptfoo (YAML, any language)

Promptfoo treats your app as a "provider" and describes tests in YAML. A Python provider is just a function called call_api:

# bot.py ANSWERS = { "What is your refund window?": "30 days", "Do you ship to Canada?": "Yes, we ship to Canada in 5-7 business days.", } def call_api(prompt, options, context): question = context["vars"]["question"] return {"output": ANSWERS.get(question, "Sorry, I don't know.")} 

# promptfooconfig.yaml description: Support bot checks prompts: - "{{question}}" providers: - file://bot.py tests: - vars: question: What is your refund window? assert: - type: equals value: 30 days - vars: question: Do you ship to Canada? assert: - type: regex value: '\d+-\d+ business days' - vars: question: Can I pay with crypto? assert: - type: not-contains value: "I don't know" 

Run it with npx promptfoo@latest eval. Result: 2 passed, 1 failed. The failure is deliberate: the bot has no answer about crypto, and the test says "never fall back to I don't know". That is exactly the kind of gap you want a test suite to surface before users do.

What stands out:

• The tests are data, not code. A QA engineer, a PM or a prompt writer can add a case without touching Python.
• It is language-agnostic. The provider could be an HTTP endpoint, a shell script or a hosted model, so it fits teams that aren't Python-first.
• It scales sideways. Add a second provider and the same tests compare two models or two prompt versions in one run. Promptfoo also has a red-teaming mode for prompt injection and jailbreak checks (not used here, because it needs a real model).

Version 2: DeepEval (Pytest, Python-native)

DeepEval looks like ordinary Pytest. Each test builds an LLMTestCase and checks it against one or more metrics:

# test_bot.py from deepeval import assert_test from deepeval.test_case import LLMTestCase from deepeval.metrics import ExactMatchMetric, PatternMatchMetric def support_bot(question: str) -> str: answers = { "What is your refund window?": "30 days", "Do you ship to Canada?": "Yes, we ship to Canada in 5-7 business days.", } return answers.get(question, "Sorry, I don't know.") def test_refund_window_exact(): q = "What is your refund window?" case = LLMTestCase(input=q, actual_output=support_bot(q), expected_output="30 days") assert_test(case, [ExactMatchMetric()]) def test_shipping_mentions_days(): q = "Do you ship to Canada?" case = LLMTestCase(input=q, actual_output=support_bot(q)) assert_test(case, [PatternMatchMetric(pattern=r".*\d+-\d+ business days.*")]) 

Run it with deepeval test run test_bot.py. Result: 2 passed.

What stands out:

• It lives next to your code. If your app is Python, tests sit in the same repo, run in the same CI job and fail the build the same way any Pytest test does.
• The real strength is the metric library. The two metrics above are simple, deterministic ones. With a model configured, DeepEval adds LLM-judged metrics for things like answer relevancy, faithfulness and hallucination, which is where RAG and agent testing gets interesting.

So which one?

After running both on the same app, the split is clearer than most comparison tables make it:

• Choose Promptfoo if you compare models or prompts, want non-developers to write tests, need red teaming, or your stack isn't Python.
• Choose DeepEval if your app is Python, you want LLM tests as Pytest quality gates in CI, or you are evaluating RAG pipelines and agents.
• Use both if you can afford the setup: Promptfoo for security and model selection, DeepEval for quality regression in CI. They don't conflict.

One honest caveat: this experiment uses a stand-in bot and deterministic checks only. The AI-judged metrics, Promptfoo's red-team generator and both dashboards need a real model and API spend, so I haven't covered them here.

The full comparison, with a feature table, CI/CD setup, costs and recommendations by role, is on my site: Promptfoo vs DeepEval: Which LLM Evaluation Framework Should You Choose?

What are you using to test your LLM features today? I'm curious whether teams lean to one tool or run both.