Before Your Team Standardizes on a Free AI Coding Stack, Run a Two-Endpoint Swap Test

Twenty tasks. That is the smallest suite I would trust to decide whether a team standardizes on a free AI coding stack — and most teams that ask me the question have zero.

Here is a pattern I keep hearing from platform leads (a composite, not one company): a free tier lands, someone runs three prompts against their own repo, two diffs look fine, and a Slack thread turns into a proposal. Then a manager asks the only question that matters — "what breaks if we're wrong?" — and nobody has a number.

So let's get the number first. Not a benchmark score from a vendor blog. A pass rate measured on your repo, with your tests as the grader.

The decision is a substitution decision, not a price decision

You are not buying tokens. You are substituting one endpoint for another in a workflow that already exists. That reframing kills most of the debate.

Ask three questions in this order:

1. Can the substitute finish the tasks I actually assign? Measured as a pass rate on your own test suite, not a public leaderboard.
2. What does a rejected diff cost me? Rejections are not free; they consume review attention and then get rewritten by hand.
3. Can I leave? If replaying your prompt suite on another endpoint takes a week, you have a lock-in problem disguised as a free trial.

Number 1 is the gate. Numbers 2 and 3 decide the shape of the rollout.

Build the swap harness before you argue about vendors

The artifact is boring and that is the point: one script, one task file, one grader you already trust. It should run against any OpenAI-compatible base URL so that switching endpoints is an environment variable, not a rewrite.

Here is the skeleton I hand to teams. I have not run it against your repo — treat it as a starting point you adapt, not a validated tool.

#!/usr/bin/env python3 """swap_harness.py: grade any OpenAI-compatible endpoint on your own tasks. tasks.jsonl, one JSON object per line: {"id":"t01","fixture":"fixtures/t01","prompt":"...","verify":"python -m pytest -q -x tests"} Run: OPENAI_BASE_URL=... OPENAI_API_KEY=... SWAP_MODEL=... \  python swap_harness.py --suite tasks.jsonl --out results_a.json """ import argparse, json, os, re, shutil, subprocess, tempfile, time from pathlib import Path from openai import OpenAI DIFF = re.compile(r"``` (?:diff|patch)\n(.*?) ```", re.S) def ask(client, model, prompt, timeout=120): t0 = time.monotonic() r = client.chat.completions.create( model=model, messages=[{"role": "user", "content": prompt}], temperature=0, timeout=timeout, ) return r.choices[0].message.content or "", time.monotonic() - t0 def run(cmd, cwd): p = subprocess.run(cmd, shell=True, cwd=cwd, capture_output=True, text=True) return p.returncode, (p.stdout + p.stderr)[-2000:] def one(client, model, task, workdir): ws = Path(tempfile.mkdtemp(dir=workdir)) / task["id"] shutil.copytree(task["fixture"], ws) text, secs = ask(client, model, task["prompt"]) m = DIFF.search(text) if not m: return {"id": task["id"], "applied": False, "passed": False, "secs": secs} patch = ws / "_swap.patch" patch.write_text(m.group(1)) code, log = run(f"git apply --whitespace=nowarn {patch}", ws) if code != 0: return {"id": task["id"], "applied": False, "passed": False, "secs": secs, "log": log} code, log = run(task["verify"], ws) return {"id": task["id"], "applied": True, "passed": code == 0, "secs": round(secs, 1), "log": log if code else ""} def main(): ap = argparse.ArgumentParser() ap.add_argument("--suite", required=True) ap.add_argument("--out", required=True) ap.add_argument("--runs", type=int, default=1) a = ap.parse_args() client = OpenAI(base_url=os.environ["OPENAI_BASE_URL"], api_key=os.environ["OPENAI_API_KEY"]) model = os.environ["SWAP_MODEL"] tasks = [json.loads(l) for l in open(a.suite) if l.strip()] rows = [] with tempfile.TemporaryDirectory() as tmp: for t in tasks: for _ in range(a.runs): rows.append(one(client, model, t, tmp)) passed = sum(r["passed"] for r in rows) summary = {"model": model, "n": len(rows), "pass_rate": round(passed / len(rows), 3)} json.dump({"summary": summary, "rows": rows}, open(a.out, "w"), indent=2) print(summary) if __name__ == "__main__": main() 

Three practical notes from people who have run variants of this:

• Use --runs 3 at temperature 0 anyway. Endpoints are nondeterministic under load; a single run measures luck.
• Keep fixtures small. A 40-file fixture makes setup failures look like model failures. Two failing tests in one module is enough.
• git apply needs a repo. Either git init inside each fixture or switch to patch -p1 --dry-run first so you can tell "bad diff" from "bad patch application".

The output is a JSON file per endpoint. Diff them, and you have a substitution argument with a receipt.

Do the rework arithmetic out loud

Pass rate alone doesn't decide anything. Convert it into minutes per merged diff:

effective minutes per merged diff = (M + (1 − P) × R) / P

Where M is touch time on a passing diff (prompt, review, small fixes), P is the measured pass rate, and R is the minutes lost on a rejected diff before you abandon it.

Illustrative numbers — not measurements from your team: M = 12 min, R = 25 min.

Now add a paid lane with M = 10 and P = 0.88: (10 + 0.12 × 25) / 0.88 = 14.8 minutes. The free lane at P = 0.80 loses by six minutes per merged diff, and no amount of "it's free" fixes that, because your reviewers are the scarce resource, not the tokens.

Run the sensitivity honestly: in that setup, there is no rework value R that makes the free lane win. It is worse on both inputs. If that is your result, say it out loud, because the free lane may still deserve a role — just not the one people were arguing for.

Where a free stack actually fits

Free model access plus a free server option (operator-supplied availability; confirm current quotas, specs, and terms yourself before you depend on them) tends to win in two places the arithmetic above misses:

• Experiment volume. Fifty throwaway probes on an unfamiliar codebase, none of which merge. Zero marginal cost matters when P is irrelevant because nothing gets reviewed.
• Procurement friction. A pilot that needs no purchase order can start Monday. That is a schedule argument, not a quality argument — and it expires the moment you ask the endpoint to produce mergeable diffs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The project offers free model access and a free server option, which makes it a reasonable first target when you want somewhere to point OPENAI_BASE_URL and start collecting your own numbers instead of reading someone else's.

The fit table I would actually sign

This table is a conversation tool, not objective truth. Two teams with the same numbers can reasonably decide differently, because their reviewers, data classes, and release cadence differ.

Who should not do this

• Teams with under 20 gradable tasks. You cannot measure substitution. Build tests first; adopt later.
• Regulated or customer-data workloads where you cannot enumerate what leaves the network.
• Anyone whose reviews are already the bottleneck. A free lane that raises your diff volume makes the actual constraint worse.

Add an expiry

Write down the decision, the pass rate that justified it, and a date. Re-run the harness at 30 and 90 days. Endpoints drift, your repo drifts faster, and a pass rate from last quarter is folklore.

So: which number would change your mind — a pass rate two points below the incumbent, or a rework cost that turns each rejection into a forty-minute archaeology session? Measure that one first.