Your API Gateway Rate Limit Can't See the Agent Looping Your Search Tool

Here's the failure mode nobody's gateway catches.

An agent gets stuck. Not stuck like a crash — stuck like a loop: it calls web_search, gets results, doesn't like them, calls web_search again with a slightly different query. 10 times. 50 times. 400 times a minute. Each call is a perfectly valid HTTP request. Your API gateway sees normal traffic well under its per-key limit. Your bill disagrees.

The problem: gateway rate limits live at the HTTP layer. Agent abuse lives at the tool-call semantic layer. One HTTP request can fan out into many tool calls. One looping tool can spread its calls across many endpoints. A gateway counting bytes and paths never sees "the agent is hammering one expensive tool" — but that's exactly the unit an agent developer wants to budget.

I built the fuse that belongs at the tool-call layer.

What I built

tool-throttle — a zero-dependency Python package (stdlib only, Python ≥ 3.9) that puts a token-bucket rate limiter on MCP tool calls, per tool name. Wrap your tool dispatcher, set quotas, and over-quota calls get queued or rejected with a retry_after hint.

pip install tool-throttle 

from tool_throttle import Throttler, ThrottlerConfig, Quota, REJECT th = Throttler(ThrottlerConfig( default=Quota(calls=60, per_seconds=60), per_tool={ "web_search": Quota(calls=10, per_seconds=60), # tight leash  "expensive_*": Quota(calls=2, per_seconds=60), # glob patterns  }, on_exceed=REJECT, # or "queue": wait for a token, up to max_queue_wait )) safe_search = th.wrap("web_search", mcp_client.call_tool) safe_search("web_search", {"query": "..."}) # past quota -> ThrottledError  th.stats() # per-tool allowed / rejected / queued / waited_seconds 

Quotas live in the language of the agent — tool names and glob patterns, with per-tool bursts — not in the language of HTTP. Config round-trips through JSON so it can sit in a file next to your agent config:

cfg = ThrottlerConfig( default=Quota(calls=60, per_seconds=60), per_tool={"web_*": Quota(calls=10, per_seconds=60, burst=3)}, on_exceed="queue", max_queue_wait=30.0, ) cfg.save("throttle.json") 

There's also a CLI for quick checks and a demo:

echo '{"tool": "web_search"}' | tool-throttle check --config throttle.json echo '{"tool": "web_search"}' | tool-throttle acquire --config throttle.json tool-throttle demo # watch a looping agent get fused 

How it works

Each distinct tool name gets its own TokenBucket, sized by the first matching quota in per_tool (glob patterns, insertion order) or default. Buckets refill steadily — Quota(calls=10, per_seconds=60) sustains 10 calls/minute with a burst of up to burst (default: calls).

On quota exhaustion:

• reject — fail fast with ThrottledError, carrying a retry_after hint so the agent can back off intelligently instead of hammering blindly.
• queue — block until a token frees up, bounded by max_queue_wait; if the bound would be exceeded, fail fast instead. Queuing keeps bursty-but-legitimate agents flowing while still capping the rate.

Everything is thread-safe, time is injectable (the test suite runs against a fake clock), and there are no dependencies — it's threading, time, json, fnmatch, and argparse.

Why this isn't just "rate limiting, again"

Because the layer matters:

• Gateway limits are per key/IP. An agent looping on one expensive tool looks normal at the HTTP layer while burning money at the tool layer. tool-throttle counts tool-call intent.
• Tool granularity. You don't want the same budget for get_time and web_search. Per-tool quotas with globs (expensive_*) express what you actually fear.
• Fuse, not wall. Queuing with a bounded wait absorbs legitimate bursts instead of erroring on the first spike — it's a fuse that trips on sustained abuse, not a wall that punishes normal use.
• Observability. stats() tells you which tool the agent was abusing, with waited-time accounting. That's the number you want when you're debugging a runaway agent at 2am.

It complements gateway rate limiting; it doesn't replace it. The gateway protects your infrastructure. tool-throttle protects your wallet from your own agent.

Links

• GitHub: https://github.com/hahahahahahahahah6/tool-throttle
• PyPI: https://pypi.org/project/tool-throttle/

MIT licensed. Feedback welcome — especially war stories about agents you've caught looping.