How I built a vision-token trimmer that prices images for one provider — and a reader caught it

Ankita is my open-source desktop AI companion: it lives on your machine, works in the background while you're away, and talks back when the job is done. Long conversations are its normal state, so it has to trim its own history — and trimming has to price images correctly, or it deletes the very screenshot the user just shared.

A few weeks ago I wrote about fixing the first version of that bug: the trimmer was weighing a screenshot's base64 string as if it were text, so a 1.6 MB PNG looked like 1.6 MB of context and got deleted before the model ever saw it. The fix reads just the image header — PNG IHDR, JPEG start-of-frame, WebP VP8 chunks — and prices the picture from its pixel dimensions instead.

Then a reader named bickov read the code and broke my central claim. This post is about the hole he found, because it is still open, and I have not decided how to close it.

The formula, and where it stops being an upper bound

The estimator lives in src/core/history.mjs and takes exactly one argument: the image URL.

export function estimateImageTokens(url) { const dimensions = imageDimensions(url); if (!dimensions) return null; let { width, height } = dimensions; const fit = 2048 / Math.max(width, height); if (fit < 1) { width = Math.max(1, Math.floor(width * fit)); height = Math.max(1, Math.floor(height * fit)); } const detail = 768 / Math.min(width, height); if (detail !== 1) { width = Math.max(1, Math.floor(width * detail)); height = Math.max(1, Math.floor(height * detail)); } return 85 + 170 * Math.ceil(width / 512) * Math.ceil(height / 512); } 

That is the OpenAI high-detail recipe: fit inside 2048 pixels, resize the short side to 768, then 85 tokens plus 170 per 512-pixel tile. Its docstring calls it "intentionally an upper bound across OpenAI-compatible vision endpoints."

Take bickov's example, a 2560×1440 screenshot. Fit to 2048 on the long edge gives 2048×1152; scale the short side to 768 gives 1365×768; that is 3 × 2 tiles, so 85 + 170 × 6 = 1,105 tokens. I reproduced that number from the code, and it matches.

But Claude does not bill in tiles. It bills roughly by pixels — width × height / 750 — and reads up to a 2576-pixel long edge. The same screenshot comes out at 4,784 tokens on bickov's numbers. That is more than four times the tile estimate. The "upper bound" is only an upper bound for one provider's pricing, wearing a docstring that claims more.

Why this bites in practice

Ankita talks to models through OpenAI-compatible endpoints, and users can point those at proxies. Behind an OpenAI-compatible proxy to Claude, the trimmer believes a screenshot costs 1,105 tokens while the model is charged roughly four times that. The trimmer keeps turns that do not fit, and the failure lands wherever the provider enforces the window — a rejected request, or history shed somewhere the user did not expect. The original bug deleted images that would have fit; this one keeps images that will not. Same class of mistake, opposite direction.

What makes it awkward is that the pricing function cannot see the answer. estimateImageTokens takes only a URL. trimMessages takes messages and byte budgets, not a provider. The agent's trimHistory knows its configured client, but nothing in that path forwards a provider or model name down to the estimator. There is no argument to fix — there is plumbing missing, and the plumbing has to cross a layer that deliberately does not know about providers.

The two honest options

Option one: thread a provider argument through. trimHistory passes the configured provider to trimMessages, which passes it to estimateImageTokens, which picks the formula per provider. Precise when the configuration is true. But behind a proxy, the configuration lies: the endpoint says OpenAI-compatible and serves Claude. A provider argument that trusts the label reproduces the bug exactly where it matters most, and a per-model pricing table rots every time a provider revises its vision billing.

Option two: price for the worst case. For every image, take the maximum across the formulas you know — tile math and pixel math — and trim against that. Safe against any provider you have modelled, and it needs no plumbing and no trusted label. But it over-trims on OpenAI: images that would have fit get dropped early, which is the original bug again, milder and in the other direction. You trade correctness on one provider for safety on all of them.

My current lean is a hybrid: price by the configured provider when the endpoint is directly configured, and fall back to worst-case pricing when the request goes through a proxy or the provider is unknown. But that lean has an obvious weak spot — "directly configured" is itself a label, and I have no reliable way to tell a direct endpoint from a proxy that speaks the same wire protocol. If you can break that distinction, the hybrid collapses back into one of the two options above.

The question

So: should a trimmer that cannot trust its provider label price images for the worst case and accept over-trimming, or thread a provider argument through and accept that proxies will lie to it? And is there a third option I am missing — something that does not depend on trusting the endpoint's self-description at all?

Ankita is open source (MIT) at https://github.com/akyourowngames/A.N.K.I.T.A, and the code in question is src/core/history.mjs. If you have priced vision tokens across providers before, tell me where my reasoning breaks first. I would rather hear it from a reader now than from a rejected request later.

Tell me I'm wrong: if you run Claude behind an OpenAI-compatible proxy, what does your trimmer do — and where exactly would my hybrid break first?