Passing Alone Is Not Passing: An Order-Sensitivity Gate for Agent Patches

An agent patch can turn a green suite into a green suite that only works in one order. Single-test runs hide it, and so does the canonical file order, because both replay the same setup history. The fix is not a longer soak: it is a three-view run matrix (isolation, canonical, shuffled) plus a fixture digest that catches in-place mutation of shared fixtures. Below is the probe, the classifier, the merge gate, and the cases where you should skip all of it.

1. The failure mode this catches

Agent patches rarely break a test in isolation. They break the contract between tests: a module-level cache that is now populated by an earlier test, an autouse fixture the patch started mutating, a os.environ write that no longer gets restored, a database row that survives teardown because the new code path skips it.

Each of those produces the same dishonest signal: pytest tests/test_orders.py::test_total passes, the file passes, CI passes — and some unrelated test three files later starts failing two days later on a different machine.

Order sensitivity is cheap to measure and expensive to guess. So measure it.

2. The three-view matrix

For the same commit, run the suite three ways:

1. Isolation: one test per process (pytest -p no:randomly <nodeid> in a loop, or --forked if you have it). Slow, but it is the only view that proves a test can stand alone.
2. Canonical: the normal collection order, exactly what CI runs.
3. Shuffled: the same suite under K different collection seeds. Start with K=3; raise it only when the cheap matrix already finds something.

The three views give you a classification, not a verdict:

Note what is not in the table: "quarantine". Quarantine is the last row of the decision table in section 5, not the first response.

3. The artifact: a probe that also fingerprints fixtures

Order shuffling alone tells you that state leaked. The digest tells you which fixture leaked it, which is the difference between a 20-minute fix and an afternoon of git bisect. This plugin is written against pytest's documented hook names; validate it on your suite before you gate merges on it.

# conftest.py — order-sensitivity probe import hashlib, json, os, random, time import pytest REPORT = os.environ.get("ORDER_REPORT", "order-report.json") _digests: dict = {} # nodeid -> {fixture_name: (obj, digest_before)} _records: dict = {} def _digest(obj) -> str: try: blob = json.dumps(obj, sort_keys=True, default=repr) except Exception: blob = repr(obj) return hashlib.sha256(blob.encode()).hexdigest()[:12] def pytest_addoption(parser): parser.addoption("--order-seed", action="store", default=None, type=int) def pytest_collection_modifyitems(config, items): seed = config.getoption("--order-seed") if seed is None: return # canonical view: do not touch order  random.Random(seed).shuffle(items) @pytest.hookimpl(hookwrapper=True) def pytest_fixture_setup(fixturedef, request): outcome = yield value = outcome.get_result() # only fingerprint payloads we can stable-serialize  if isinstance(value, (str, int, float, bool, list, dict, tuple, type(None))): _digests.setdefault(request.node.nodeid, {})[fixturedef.argname] = (value, _digest(value)) def pytest_runtest_setup(item): _records[item.nodeid] = { "order_seed": item.config.getoption("--order-seed"), "started": time.time(), } def pytest_runtest_logreport(report): if report.when == "call": _records.setdefault(report.nodeid, {})["outcome"] = report.outcome def pytest_runtest_teardown(item, nextitem): rec = _records.setdefault(item.nodeid, {}) mutated = [] for name, (value, before) in _digests.get(item.nodeid, {}).items(): after = _digest(value) # same object, re-hashed  if after != before: mutated.append({"fixture": name, "before": before, "after": after}) rec["mutated_fixtures"] = mutated rec["duration_s"] = round(time.time() - rec.get("started", time.time()), 3) def pytest_sessionfinish(session, exitstatus): with open(REPORT, "w") as fh: json.dump(_records, fh, indent=2, sort_keys=True) 

Run the three views:

# canonical ORDER_REPORT=canonical.json pytest -q # shuffled, three seeds for seed in 1 2 3; do ORDER_REPORT="shuffle-$seed.json" pytest -q --order-seed "$seed" done # isolation: one nodeid per process, emitted by --collect-only pytest --collect-only -q | head -n -2 | while read -r node; do ORDER_REPORT="iso-$(echo "$node" | tr '/:' '__').json" pytest -q "$node" done 

The fixture digest is the part that earns its keep. mutated_fixtures non-empty means a test changed an object the fixture handed out, and the next test inherits that change. That is a defect in the test or the patch, never in the runner.

4. Classify and gate in one script

# classify.py — merge the matrix, print a table, return an exit code import json, sys def load(path): with open(path) as fh: return json.load(fh) iso, canon = load(sys.argv[1]), load(sys.argv[2]) shuffles = [load(p) for p in sys.argv[3:]] def out(rep, node): return rep.get(node, {}).get("outcome", "missing") rows = [] for node in sorted(set(canon) | set(iso)): o_iso, o_can = out(iso, node), out(canon, node) o_shuf = [out(r, node) for r in shuffles] mutated = bool(canon.get(node, {}).get("mutated_fixtures")) if mutated: klass = "fixture-mutation" elif o_iso == "failed" and o_can == "passed" and all(o == "passed" for o in o_shuf): klass = "setup-dependent" elif "failed" in o_shuf and o_can == "passed": klass = "order-sensitive" elif o_can == "failed": klass = "seed-or-patch" else: klass = "stable" rows.append((node, klass, o_iso, o_can, "".join("P" if o == "passed" else "F" for o in o_shuf))) for r in rows: print(f"{r[0]:52} {r[1]:17} iso={r[2][:4]:4} can={r[3][:4]:4} shuf={r[4]}") block = {"fixture-mutation", "order-sensitive"} signal = {r[1] for r in rows} & block if signal: sys.exit(1) # block the patch if any(r[1] == "seed-or-patch" for r in rows): sys.exit(2) # investigate before merge sys.exit(0) 

python classify.py iso.json canonical.json shuffle-1.json shuffle-2.json shuffle-3.json # exit 1 -> shared-state defect introduced by the patch # exit 2 -> genuine failure or nondeterminism; rerun before deciding anything 

5. The decision table the exit codes encode

A freeze is a loan with an interest rate. If you freeze, write the record — and require the class field, so a freeze cannot be filed without evidence:

{ "nodeid": "tests/test_orders.py::test_total", "class": "order-sensitive", "evidence": "shuffle-2.json#tests/test_orders.py::test_total", "owner": "payments-team", "expires": "2026-12-06", "exit_criteria": "10 consecutive shuffled seeds pass in CI" } 

Two rules keep this honest. A freeze without a class is rejected by the gate. A freeze with an expired expires fails the build the same way a real failure does — otherwise the ledger becomes a graveyard.

6. Triage where the free tier is actually useful

Shuffled runs produce boring, repetitive output: twenty tests with the same P/P/F signature and the same mutated fixture. That is a clustering problem, not a reasoning problem, which is the part of the workflow where MonkeyCode's free model access fits — feeding the classifier JSON and asking for failure-signature clusters and a minimal two-test reproduction order. The free server option is the other half: the matrix is K + 2 full suite runs, so it belongs on a runner rather than on your laptop. Availability of both is operator-supplied; I have not benchmarked throughput, so time one full run of your own suite and multiply before committing to K=10.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The probe, the classifier, and the decision table stand on their own. If you remove every product mention, the section 5 table still blocks the same patches.

7. Limitations, honestly

• Digests only cover serializable fixtures. A fixture returning a live DB connection is invisible to _digest. For those, fingerprint rowcount or a SELECT count(*) snapshot instead.
• Shuffling is sampling, not proof. Passing 3 seeds does not prove order independence; it bounds your confidence. Do not convert a green shuffle matrix into a claim of correctness.
• Isolation runs are expensive. Suites with session-scoped migrations can go from 90 seconds to 20 minutes. Run the isolation view nightly, not per PR.
• External services masquerade as order sensitivity. Two tests hitting a shared rate-limited sandbox will classify as order-sensitive while having nothing to do with order. Check duration_s and retry the shuffled view once before believing it.
• setup-dependent is a design smell, not a bug. Plenty of suites legitimately chain state. Annotate those tests rather than rewriting them, or you will fight your own test suite instead of the patch.

8. Who should not use this

Skip it if your full suite takes more than roughly ten minutes: K + 2 runs turn a PR check into an hour. Skip it if you cannot store report artifacts next to the build — the classification is worthless without the evidence file it points to. Skip it if your tests intentionally share a mutable session fixture as a performance strategy; the fixture digest will fire on every test and you will learn to ignore it, which is worse than not having it.

Otherwise, adopt in this order: fixture digest first (one hook, catches the loudest defects), canonical vs shuffled second (no isolation runs yet), isolation third (nightly only). Each step is independently useful, and each one makes the merge gate a little less of a coin flip.