Our Tool Makes AI Answers Up to 18x Pricier. I Published It Anyway.

2026-11-18来源 dev.toEnglish

I asked for the cost numbers expecting a post that said our tools knock cost down. Then the numbers came back. With our tools, an answer cost between 4 and 18 times as much.

This is about what I did with that, and two smaller surprises in the same run that taught me more than the headline did.

I build Since, which tracks the packages, AI models, APIs and runtimes your code depends on, tells you and your coding agent what changed and when it breaks, and opens the fix pull request. Part of that is two tools a coding agent can call before it writes code: whats_changed (what changed in this model, package or runtime since a date) and check_code (what in this code is retired, ending or vulnerable).

1. The experiment

The question was simple. How often does a coding model write code that is already dead on the day it writes it? A retired model ID, a runtime past end of life, a pinned version with a known advisory.

We wrote 60 everyday prompts in 7 areas. "Write a Python function that summarizes a customer support ticket in three bullet points using the Anthropic SDK." "Write the package.json for a new Next.js app with Tailwind CSS, NextAuth and Stripe. Pin exact versions." The kind of thing a coding agent gets every day.

Nine models answered every prompt twice. Once on their own. Once with our two tools and one rule: read what's current before you write a model ID or a version, and check the code before you answer.

Every code block in every answer went through the same check anyone can call, and each answer was counted once, by the worst thing in it.

The run finished on November 10: 1,080 answers, 0 errors, about $64 at list prices.

2. Zero errors, and the judge was reading page one

The first surprise came before any headline. "0 errors" felt like a clean run. It wasn't a checked run.

The judge reads findings that come back in pages, most severe first. It was reading only the first 500 findings of a block. For most answers that was everything. For some it wasn't, and severity isn't the order the judge needs.

We changed it to read every page and re-judged the 1,080 stored answers. That cost nothing, because the answers were already saved: no model had to answer again. 26 verdicts changed. 14 went from "behind" to "vulnerable", 12 from "clean" to "behind". None moved into or out of dead or ending, so the headline numbers held. But a run that finishes with no errors only tells you nothing crashed. It doesn't tell you anything was read.

3. Our own tools were weaker than our product

The second surprise was more embarrassing. In the "with Since" half, the tools the models got were worse than what we actually ship.

whats_changed was passing a package name where it expected a model. So 1,741 of its 2,922 calls came back empty. The model asked a good question and got silence. And check_code answers were cut off at 3,000 characters, so a long answer lost its tail.

Both are fixed for later runs. A trial on 12 answers with the fixed tools: 12 clean, against 6 before, at the same cost.

We published the November 10 run anyway, with the half-broken tools, because that is what ran. Even so, with the tools, the share of answers that were dead or ending dropped to between 0% and 6.8%, from between 16.9% and 44.1% on their own.

So far, a good story. Then I asked for the cost.

4. The number that went the wrong way

I asked for tokens and cost, with Since and without, fully expecting to post that the tools cut cost.

I hadn't thought it through. The benchmark measures one request. And for one request, a model with tools reads what the tools answer. That is a lot of text.

Here's the check I ran on the published table:

# From the "What the tools cost" table on since.dev/dead-on-arrival (run of Nov 10, 2026): # tokens alone, tokens with the tools, cost alone, cost with the tools, per answer. rows = { "model A": (577, 33721, 0.0049, 0.086), "model B": (1061, 20980, 0.0037, 0.021), "model C": (1454, 17334, 0.0007, 0.0026), "model D": (1434, 23154, 0.013, 0.062), "model E": (924, 18454, 0.0004, 0.0027), "model F": (2315, 51518, 0.089, 0.56), "model G": (1391, 21216, 0.026, 0.12), "model H": (841, 12262, 0.0039, 0.017), "model I": (776, 15805, 0.0071, 0.041), } tok = [w / a for a, w, _, _ in rows.values()] cost = [cw / ca for _, _, ca, cw in rows.values()] print(f"tokens: {min(tok):.1f}x to {max(tok):.1f}x") print(f"cost: {min(cost):.1f}x to {max(cost):.1f}x") print(f"cheapest with tools vs priciest alone: {0.089 / 0.0026:.0f}x cheaper") 

What it printed:

tokens: 11.9x to 58.4x cost: 3.7x to 17.6x cheapest with tools vs priciest alone: 34x cheaper 

12 to 58 times the tokens. 4 to 18 times the cost. Per answer, our tools make every model more expensive. There is no way to read that table that says otherwise.

5. The claim the data actually supports

I had two options. Leave the cost table off the page, or put it on and find what's true.

It went on the page, under the heading "What the tools cost", with the line "The tools are not free." Then I looked at the same table from the other side, and the last line of that script is the answer.

The cheapest model in the run, with our tools, cost $0.0026 an answer, and 1.7% of its answers were dead or ending. The most expensive model, on its own, cost $0.089 an answer, and 28.8% of its answers were.

So the honest claim isn't "Since makes your model cheaper." It's closer to: a small model that can check what's current beats a big model that's guessing, at about a thirtieth of the price. That's a better claim than the one I wanted. It's also one nobody can knock down by opening the table.

6. The same week, I almost paid to make a table prettier

A smaller version of the same lesson, three days later. I needed before-and-after numbers for some work on our tool descriptions. We had them, with one soft spot: for two models, the "before" was a rewrite from that morning, not the text that had been live.

So 11 more paid eval runs started, to make the table cleaner. I stopped them. The numbers we had were real; they just needed one honest sentence saying what the "before" was. A paid run should exist because a decision depends on it, not because a write-up would look neater.

When this is the wrong call

Putting an unflattering number on your own page isn't always right, and adding tools to a model isn't always worth it.

  • If your request never touches anything with a date on it, the tools are pure cost. A function that sorts a list doesn't need to know what retired last week.
  • If you're paying per request at high volume and latency matters more than correctness, 12 to 58 times the tokens is a real bill. Check what fraction of your requests write model IDs or versions at all.
  • If your sample is small, say so. Each model answered each prompt once. A rate from 60 answers moves a few points between runs, so a difference of one or two answers isn't a finding. That line is on the page too.

What I took from it: compute the number before you write the sentence around it. I had the sentence first, and the number disagreed.

What's a metric you were sure would make your work look good, until you actually measured it?

I'm Nabil. I build Since, which tracks what your code depends on and tells you before it breaks.

原文:https://www.a8q9.cn/en/article/our-tool-makes-ai-answers-up-to-18x-pricier-i-published-it-anyway-310d80.html

相关文章