A Passing Characterization Test Proves Nothing Until You Break the Code

Most characterization tests in a messy repo are decoration. They pass before the refactor. They pass after it. They would also pass if the branch under test were deleted.

That is not a test. It is a receipt. This post adds one cheap gate: prove each new test can fail, by breaking the code on purpose.

The failure mode: tests that assert nothing

A vacuous test usually appears during refactor prep. Someone snapshots a return value. Or writes assert result is not None. It goes green, the team feels safe, and the gate stays open.

Then the refactor changes rounding, ordering, or a default. The suite stays green. The bug ships. The test was never guarding the seam you touched.

Coverage does not catch this. A line can execute and still be unobserved. The only honest question is: which code change makes this test fail?

The rule: one test, one kill

Adopt a single rule for refactor prep. Every new characterization test must kill at least one deliberate mutation.

A mutation is a tiny, mechanical edit to the source. Change >= to > — swap a default — drop a rounding call. If the test still passes, the test cannot see that behavior.

Keep the mutant set small and hand-written. You are guarding a seam, not scoring the whole file.

Step 1: Pick the seam, not the file

Refactors fail at boundaries, not at file scope. Name the boundary before you write anything: a function, a public method, a serialized payload.

Write that name down. Also write the caller that must keep working. If you cannot name the caller, you are not ready to refactor.

Step 2: Freeze every input you cannot control

Mutation results are worthless when the test is random. Pin time, IDs, and env reads behind module-level functions you can patch.

# tests/conftest.py import pytest from billing import prorate @pytest.fixture(autouse=True) def frozen_inputs(monkeypatch): monkeypatch.setattr(prorate, "_now", lambda: "2026-03-01T00:00:00Z") monkeypatch.setattr(prorate, "_new_id", lambda: "id-fixed-001") monkeypatch.setenv("BILLING_ROUNDING", "half_up") 

Run the target test five times in a row. If any run differs, stop. Fix determinism before you fix anything else.

Step 3: Capture real bytes, then pin them

Run the current code and record what it actually produces. Not what you believe it should produce. Paste that output into the assertion.

def test_partial_month(): out = prorate.compute(days_left=1, cents=1234) assert out == {"amount_cents": 1234, "label": "1 day"} 

Ugly golden values are fine here. They are temporary scaffolding for the diff. You delete or re-derive them after the behavior is documented.

Step 4: Write mutants by hand

List the edits that would break a caller without crashing. Boundaries, defaults, rounding, and ordering are the usual suspects.

Keep them as data, not as a tool config. Two to six entries per seam is enough.

MUTANTS = [ {"id": "M1", "file": "billing/prorate.py", "old": "if days_left >= 1:", "new": "if days_left > 1:", "target": "tests/test_prorate.py::test_partial_month"}, {"id": "M2", "file": "billing/prorate.py", "old": "return round(cents, 2)", "new": "return round(cents, 0)", "target": "tests/test_prorate.py::test_partial_month"}, ] 

Each mutant must match exactly once. A pattern that matches zero or three times is a broken mutant, not a survived one.

Step 5: Run the loop

The runner copies the repo, applies one mutant, runs one test node, then restores the file. Classify the exit code honestly.

# mutate.py — unexecuted scaffolding; adapt paths and selectors first. import shutil, subprocess, sys, tempfile from pathlib import Path from mutants import MUTANTS REPO = Path(".").resolve() def classify(code: int) -> str: if code == 1: return "killed" # assertion failed: the test saw it  if code == 0: return "survived" # test ignored the change  if code == 5: return "no-tests" # selector matched nothing  return "error" # 2 interrupted, 3 internal, 4 usage  def run(mutant): with tempfile.TemporaryDirectory() as tmp: work = Path(tmp) / "repo" shutil.copytree(REPO, work, ignore=shutil.ignore_patterns(".git", "__pycache__")) src = work / mutant["file"] original = src.read_text() if original.count(mutant["old"]) != 1: return mutant["id"], "bad-pattern" src.write_text(original.replace(mutant["old"], mutant["new"])) proc = subprocess.run( [sys.executable, "-m", "pytest", "-q", "--no-header", mutant["target"]], cwd=work, capture_output=True, text=True, timeout=300, ) return mutant["id"], classify(proc.returncode) failed = False for m in MUTANTS: mid, verdict = run(m) print(f"{mid:<4} {verdict:<12} {m['target']}") failed |= verdict in {"survived", "no-tests", "bad-pattern", "error"} sys.exit(1 if failed else 0) 

Exit code 5 deserves attention. It means the selector matched no tests, which silently poisons the whole gate.

Step 6: Read the results like a table

Survivors split into two kinds. Real gaps get a tighter assertion. Equivalent mutants — edits with no observable behavior change — should be deleted from the list, not chased.

Where a free model helps, and where it does not

Writing mutants by hand is fast. Finding the seams you forgot is not. A model is decent at proposing candidate assertions and candidate mutants for unfamiliar code.

Give it the function and the callers. Ask for boundary inputs and, for each input, the one-line edit that should change the output. Then run the loop.

The loop is the judge. A model proposal that kills nothing is dropped, no matter how plausible the reasoning sounds. Free model access on MonkeyCode is enough for this propose-then-verify pattern, since the artifact is text, not a passing test suite. The same loop runs fine on a remote machine; MonkeyCode's free server option is one place to park it if you would rather not pin a laptop core. Check the current quotas and limits yourself — free tiers change without notice.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Step 7: Make the smallest safe change

Now edit production code. One seam per commit. Then re-run the exact same mutant set.

The kill set must be identical before and after. New kills mean you changed behavior nobody asked for. Lost kills mean the test stopped guarding the seam.

That comparison is the gate. It is also cheaper than reviewing a large diff by eye.

Limitations and who should skip this

• Mutation runs are slow. One test node per mutant keeps it tolerable; whole-suite runs do not.
• Textual mutants can produce syntax errors. Treat those as harness bugs, not kills.
• Equivalent mutants waste time forever. Delete them aggressively and note why.
• This proves tests observe behavior. It does not prove the behavior is correct.
• Skip it during rewrites. If the seam is being replaced, characterization is the wrong tool.
• Skip it if a single test run already takes minutes. Fix runtime first.

Start with one module and one seam. Run the loop once. Count the survivors before you write any new test.

That count is your real starting point for the refactor.