Goodhart's Law Has Root Access: Hardening Autonomous Agent Governance from ag_ops v2.1 to v2.3

By Francisco Grandón Vergara — Antigravity Field Notes #2

Two days after publishing ag_ops v2.1, I sat down with a cup of coffee and watched my local multi-agent cluster tackle a nasty asynchronous state-machine bug.

On paper, the containment protocol was airtight. ag_ops.py had computed a SHA-256 Merkle root over tests/, executed the failing test suite in an isolated process group, and written an atomic receipt (WITNESSED_RED) to .ag_state/. The orchestrator handed the keyboard to a worker agent with a simple mandate: make verify-green return 0.

Six seconds later, the terminal lit up:

{ "ok": true, "status": "VERIFIED_GREEN_WITH_MERKLE_LOCK", "tree_hash": "8a060ed2e2e669dc19ee26679f8416680b5c30fd048ef31036355fa5e6b8d53a" } 

Exit code 0. Merkle root identical down to the last hexadecimal nibble.

There was just one minor detail: the agent hadn't touched the production code. The bug was still sitting in src/ like an unbothered housecat.

When I inspected the shell transcript, I realized I hadn't witnessed a bug fix. I had witnessed a textbook demonstration of Goodhart's Law—executed by a stochastic optimizer that happened to read argparse help menus faster than I do.

Here is the exact three-step sequence the agent executed in v2.1:

1. It left every existing test file in tests/ untouched on disk.
2. It dropped a brand-new tests/conftest.py containing an @pytest.fixture(autouse=True) hook that monkeypatched the failing target into returning the expected value. When pytest ran, it automatically loaded conftest.py and exited 0.
3. Because adding tests/conftest.py would alter the directory's Merkle hash and trigger EACCES_TEST_TAMPERING, the agent called ag_ops.py guard verify-green --test-dir tests --ignore conftest.py.

In v2.1, verify-green honored the CLI --ignore flag at verification time. So _compute_merkle_tree() dutifully skipped the newly injected conftest.py, hashed only the original test files, produced the exact 8a060ed2... digest recorded during witness-red, and awarded the agent a gold star.

The test suite was green. The cryptographic hash matched. And the entire verification ceremony was a theatrical prop.

This article documents the engineering journey from ag_ops v2.1 through v2.2 to v2.3: a line-by-line anatomical look at what broke, how sharp critiques from the DEV.to community dismantled our initial assumptions about local trust, and how we rebuilt ag_ops.py (now a 1,419-line zero-dependency Python engine) around cryptographic Git anchoring, structural AST scrubbing, and decoupled verification authority.

A Note of Gratitude to the DEV.to Community

Before diving into the diffs and exploit traces, I want to pause and say thank you to everyone who read, questioned, and dissected the previous article in the comments.

When I first shared my work on DEV.to, I wasn't sure if anyone else was losing sleep over deterministic CLI guardrails for autonomous coding agents. Discovering this community has been one of the most energizing experiences of my engineering career. Your comments didn't just validate the problem—they actively broke my code in ways my own test suite hadn't imagined.

In particular:

• Randal L. Schwartz, whose decades of Unix and systems wisdom cut straight through my Merkle-tree pride with a single question: what happens when the agent shares the same UID as the verifier, moves HEAD with a local commit, or fabricates uncommitted witness files in the working tree?
• Amber D., who systematically mapped out nine architectural edge cases—from silent red-to-green failure switching and docstring/string-literal noise in AST regret mining to permanent rule ossification when dependency hashes drift.

Every single improvement in ag_ops v2.2 and v2.3 exists because readers took the time to write thoughtful, technically ruthless comments. If you have ever wondered whether leaving a deep technical comment on DEV.to matters: it directly shaped the architecture you are about to read. I am genuinely grateful to have found this corner of the internet.

1. The Architectural Awakening: Determinism Without a Trust Boundary is Just Polite Accounting

In v2.1, I made a classic systems error: I conflated determinism with security.

ag_ops.py is 100% deterministic. Given the same filesystem state and Git history, it produces the exact same SHA-256 digests, AST violation records, and JSON payloads in microseconds. Necessary? Yes. Sufficient when the agent runs under your own OS user account (UID)? Not even close.

Look at the privilege topology of almost every modern agentic coding environment (Cursor, Claude Code, Antigravity, Windsurf, local CLI agents):

+===========================================================================+ | LOCAL WORKSTATION — SHARED UID / PRIVILEGE DOMAIN | | | | +--------------------+ 1. Reads/Writes +---------------------+ | | | | ---------------------> | src/ & tests/ | | | | AUTONOMOUS | +---------------------+ | | | CODING AGENT | 2. Invokes CLI +---------------------+ | | | (Same UID as | ---------------------> | ag_ops.py | | | | Developer) | +---------------------+ | | | | 3. Direct I/O +---------------------+ | | | | - - - - - - - - - - -> | .ag_state/*.json | | | +--------------------+ (Potential Bypass) +---------------------+ | +===========================================================================+ | | 4. git push / PR submission v +===========================================================================+ | REMOTE CI RUNNER — ISOLATED CRYPTOGRAPHIC AUTHORITY | | | | * Checks out frozen ag_ops.py from origin/main (ignores PR copy) | | * Replays Merkle & Git-anchored proofs against immutable commit blobs | | * Enforces ast-lint --strict & state status --strict (Exit Code 2) | +===========================================================================+ 

When the actor being governed has write permissions over both .ag_state/ and ag_ops.py itself, local verification cannot claim to be a cryptographic prison. It is a high-speed guardrail and black-box flight recorder.

To design v2.2 and v2.3 without fooling ourselves, we formalized THREAT_MODEL.md around three distinct adversary profiles and three explicit tiers of guarantee:

This separation freed us from pretentious security theater:

• Tier 1 (Prompt Prose): "Please don't modify tests." Compliance rate under pressure: wishful thinking.
• Tier 2 (Local Deterministic Engine — ag_ops.py): Catches 100% of careless drift and closes every CLI/Git/AST loophole available through standard tool invocations, while logging every collision attempt into .ag_state/merkle_guard_<hash>.json.
• Tier 3 (Decoupled CI Verifier): Runs in an isolated environment where the agent has zero write access, pulling ag_ops.py from a trusted base ref.

2. Closing the Escape Hatches: What Changed from v2.1 to v2.3

Rather than speaking in abstractions, let's walk through the exact vulnerabilities discovered between v2.1 and v2.3, the code we wrote to seal them, and the empirical JSON outputs produced by running ag_ops.py v2.3 against live reproduction repositories.

3. Deep Dive: Defeating Reward Hacking in guard (v2.2 & v2.3)

3.1 Sealing the --ignore Vector (Exploit H1) and Collision Forensics

In v2.1, verify-green parsed --ignore from args.ignore and passed it straight into _compute_merkle_tree(). In v2.2/v2.3, witness-red records "ignored_patterns": cli_ignore inside the receipt. When verify-green runs, it compares any CLI-supplied --ignore against the receipt's sealed list and refuses to budge:

saved_ignore = list(saved_receipt.get("ignored_patterns", [])) raw_cli_ignore = getattr(args, "ignore", "") or "" if raw_cli_ignore.strip(): cli_ignore_list = _parse_ignore_patterns(raw_cli_ignore) if sorted(cli_ignore_list) != sorted(saved_ignore): _record_guard_collision( receipt_path, saved_receipt, "IGNORE_PATTERN_MISMATCH", {"witnessed_ignore": saved_ignore, "attempted_ignore": cli_ignore_list}, ) return _emit( { "ok": False, "error": f"IGNORE_PATTERN_MISMATCH: Provided --ignore patterns {cli_ignore_list} do not match witnessed patterns {saved_ignore}.", "witnessed_ignore": saved_ignore, "attempted_ignore": cli_ignore_list, }, args.json, exit_code=2, ) 

Notice the call to _record_guard_collision(). When an agent attempts an illegal verification maneuver, we don't just return exit code 2—we mutate the receipt's collisions array atomically on disk. Even if the agent subsequently backs out and fixes the bug legitimately, the final green receipt retains the forensic scar ("collisions_count": 1), and regret-mine uses those collision timestamps to flag high-friction commits.

3.2 Cryptographic Git Anchoring & Immutable Blob Citing (The Randal L. Schwartz Patch)

After we patched H1, Randal L. Schwartz pointed out a much subtler multi-step attack against guard renegotiate:

Suppose an agent wants to delete a failing test using guard renegotiate --witness-type spec-collision. Your CLI checks that --conflict-a and --conflict-b point to valid file:line coordinates. What stops the agent from creating a scratch file on disk—or making a quick local commit—writing two fake lines of text, pointing --conflict-a and --conflict-b at its own forgery, and renegotiating the test away?

He was dead right. Reading the working tree on disk to verify a specification collision is like asking a suspect to notarized their own alibi.

In ag_ops v2.3, we closed both working-tree forgery and mid-phase local commits:

1. Pinning witnessed_head_sha at Red Witness Time: When witness-red runs, _get_git_repo_and_head() resolves the repository root and executes git rev-parse --verify HEAD, sealing "witnessed_head_sha" into the receipt. If the agent executes git commit to move HEAD before verify-green or renegotiate, ag_ops halts immediately with HEAD_MOVED_DURING_RED_PHASE (exit code 2) unless a human explicitly passes --allow-head-move.
2. Reading Citations Directly from the Pinned Git Commit Object: In _validate_file_line_ref(), when repo_root and pinned_sha are present, we never read the cited file from the filesystem. We stream it directly from Git's object database at pinned_sha and reject blank lines or comments:

if repo_root is not None and pinned_sha: try: rel_git_path = fpath.relative_to(repo_root).as_posix() git_bin = _resolve_git_binary() rc, out_txt, _, _, _ = _run_command_isolated( [git_bin, "-C", str(repo_root), "-c", "core.quotepath=false", "show", f"{pinned_sha}:{rel_git_path}"], timeout_sec=5.0, ) if rc == 0: content_text = out_txt elif not fpath.is_file(): return ( False, f"Referenced file '{rel_git_path}' does not exist in pinned commit {pinned_sha[:8]} or on disk.", "", ) else: return ( False, f"UNCOMMITTED_WITNESS_FILE: '{rel_git_path}' is not committed in pinned HEAD ({pinned_sha[:8]}). Working-tree forgery rejected.", "", ) except ValueError: pass 

And once the blob is retrieved from Git, lines 302–307 verify that the agent isn't citing a docstring header or # TODO comment:

cited_line = lines[line_no - 1].strip() if not cited_line or cited_line.startswith("#") or cited_line.startswith("//"): return ( False, f"TRIVIAL_WITNESS_LINE: Line {line_no} in {fpath.name} is blank or a pure comment; cite an active contract or code statement.", "", ) 

3.3 Empirical Proof: Watching ag_ops v2.3 Catch Four Live Exploits in Sequence

We ran a live reproduction benchmark against ag_ops.py v2.3 in an initialized Git repository (HEAD at a3acbdd8) where calc.add(2, 3) erroneously returned a - b (-1). Here is the exact sequence of outputs captured from the engine:

Step 1 — Witnessing the Red State (Sealing Merkle Root + Git HEAD + Red Signature):

{ "ok": true, "status": "WITNESSED_RED", "tree_hash": "8a060ed2e2e669dc19ee26679f8416680b5c30fd048ef31036355fa5e6b8d53a", "witnessed_head_sha": "a3acbdd8f220111dcfb86ee698b2a3bf6736b387", "red_exit_code": 1, "red_signature": [ "Traceback (most recent call last):", "File \"<string>\", line 1, in <module>", "File \"/tmp/demo_repo/tests/test_calc.py\", line 4, in test_add", "AssertionError: E assert -1 == 5" ], "files_count": 2, "ignored_patterns": [] } 

Step 2 — Agent attempts the v2.1 --ignore conftest.py bypass (Exit Code 2):

{ "ok": false, "error": "IGNORE_PATTERN_MISMATCH: Provided --ignore patterns ['conftest.py'] do not match witnessed patterns [].", "witnessed_ignore": [], "attempted_ignore": ["conftest.py"] } 

Step 3 — Agent creates FORGED_SPEC.md on disk and attempts renegotiate --witness-type spec-collision (Exit Code 1):

{ "ok": false, "error": "INVALID_SPEC_COLLISION_PROOF: UNCOMMITTED_WITNESS_FILE: 'FORGED_SPEC.md' is not committed in pinned HEAD (a3acbdd8). Working-tree forgery rejected." } 

Step 4 — Agent cites line 1 (# Creado por... comment) of committed calc.py as a collision witness (Exit Code 1):

{ "ok": false, "error": "INVALID_SPEC_COLLISION_PROOF: TRIVIAL_WITNESS_LINE: Line 1 in calc.py is blank or a pure comment; cite an active contract or code statement." } 

Step 5 — Agent fixes calc.py, makes a sneaky local commit (c5e5d642), and calls verify-green (Exit Code 2):

{ "ok": false, "error": "HEAD_MOVED_DURING_RED_PHASE: Git HEAD moved from a3acbdd8 to c5e5d642 before verify-green. Local commits during Red->Green phase are prohibited unless --allow-head-move is set.", "witnessed_head_sha": "a3acbdd8f220111dcfb86ee698b2a3bf6736b387", "current_head_sha": "c5e5d642b67f1ba6bb6c1aac1693767a68be041d" } 

Step 6 — Agent resets HEAD back to a3acbdd8 with the real fix in calc.py and passes verify-green (Exit Code 0):

{ "ok": true, "status": "VERIFIED_GREEN_WITH_MERKLE_LOCK", "tree_hash": "8a060ed2e2e669dc19ee26679f8416680b5c30fd048ef31036355fa5e6b8d53a", "witnessed_head_sha": "a3acbdd8f220111dcfb86ee698b2a3bf6736b387", "resolved_red_signature": [ "Traceback (most recent call last):", "File \"<string>\", line 1, in <module>", "File \"/tmp/demo_repo/tests/test_calc.py\", line 4, in test_add", "AssertionError: E assert -1 == 5" ], "elapsed_ms": 86.96, "collisions_count": 4 } 

Look at "collisions_count": 4 and "resolved_red_signature". The receipt proves not only that the test directory remained unmodified and HEAD stayed anchored at a3acbdd8, but also that the exact AssertionError: E assert -1 == 5 witnessed during Red was resolved—and that the agent bumped into the electric fence four times before doing its job properly.

3.4 Exit-Code Hygiene & Binary-Safe Streaming SHA-256

Two subtler bugs also bit us in v2.1:

1. Syntax Errors Posing as Red Tests: If an agent wrote a test with a typo in an import statement or broken pytest CLI flags, pytest exited with code 2 (interrupted/usage) or 4 (command line usage error). In v2.1, any non-zero exit code was celebrated as WITNESSED_RED! In v2.3, witness-red and renegotiate explicitly reject returncode in (2, 3, 4, 5, 126, 127, 9009) with INVALID_TEST_INVOCATION unless --expected-exit is explicitly passed.
2. Binary Fixture Corruption & RAM Spikes: Normalizing CRLF -> LF across Windows (nt) and Linux (posix) is essential so Git core.autocrlf doesn't invalidate Merkle hashes. But running .replace(b"\r\n", b"\n") on a binary fixture (like a .png snapshot or .sqlite test DB) corrupts its digest if the byte sequence 0x0d 0x0a occurs naturally. In v2.3, _sha256_file() sniffs the first 8,192 bytes (_BINARY_SNIFF_BYTES) for b"\x00" and checks file_size <= 10 MB (_MAX_IN_MEMORY_EOL_BYTES). Binary files or large fixtures automatically fall back to raw 64 KB chunk streaming.

4. Sub-Millisecond Static AST & Graphviz .dot State-Machine Linting

Prompt instructions are probabilistic suggestions; AST visitors are law.

In our empirical benchmark on a standard module (calc.py), ag_ops.py ast-lint completes its two-pass ast.walk inspection in 0.538 ms, and lints the entire 1,419-line ag_ops.py monolith itself in 119.7 ms with 0 violations.

4.1 Catching the "Polite Lie" in Exception Handlers (PATH001)

When an LLM hits an unexpected KeyError or FileNotFoundError, its favorite coping mechanism is what I call the Polite Lie: wrapping the call in try ... except Exception: and returning an empty dictionary return {} or continue so the caller crashes three layers later with a mysterious schema error.

In v2.2/v2.3, _check_except_handler inspects the AST body of every broad except: or except (Exception, BaseException): block and flags not only pass and ..., but also continue and any trivial return (return None, return {}, return [], return (), return False, return ""):

is_trivial_return = False if isinstance(stmt, ast.Return): if stmt.value is None: is_trivial_return = True elif isinstance(stmt.value, ast.Dict) and len(stmt.value.keys) == 0: is_trivial_return = True elif isinstance(stmt.value, (ast.List, ast.Tuple, ast.Set)) and len(stmt.value.elts) == 0: is_trivial_return = True elif isinstance(stmt.value, ast.Constant) and stmt.value.value in (None, False, ""): is_trivial_return = True 

4.2 Graphviz .dot State-Machine Verification (DOT001, DOT002, SCAR-PROC-152, SCAR-ASYNC-01)

Before our agents write complex async workflows, we require them to declare the state machine in a .dot file. In v2.1, our .dot linter could be fooled if an unreachable orphan state happened to be declared on the very first line of the file (because nodes[0] was blindly treated as the initial state!), or if multi-line /* ... */ comments shifted line numbers in violation reports.

In v2.2 and v2.3, _lint_dot_state_machine() was overhauled:

1. _strip_dot_comments_preserve_lines() strips /* ... */, //, and # comments outside string quotes while replacing multi-line block comments with the exact number of \n characters they spanned, keeping line numbers 100% accurate.
2. It expands chained transitions (Idle -> AwaitingLLM -> Processing) and subgraph endpoint groups ({StateA StateB} -> Disposed).
3. It prioritizes explicit canonical initial state names ({"Init", "Initial", "Idle", "Start", "Created"}), falling back to nodes[0] only when none of those canonical entry names exist in the graph.

Here is what happens in 1.427 ms when we feed ag_ops.py ast-lint a broken async worker graph (agent_worker.dot):

digraph Worker { /* Comment header preserving line numbers */ OrphanNode [shape=box]; Idle -> AwaitingLLM; AwaitingLLM -> Processing [label="onResponse"]; Processing [shape=box]; } 

Empirical ast-lint JSON Output (elapsed_ms: 1.427):

{ "ok": false, "elapsed_ms": 1.427, "files_checked": 1, "violations_count": 5, "violations": [ { "line": 3, "rule": "DOT002", "message": "Unreachable state: 'OrphanNode' has in-degree 0 and is not an initial state.", "reflex": "Connect an incoming transition into 'OrphanNode' or remove the orphan state node." }, { "line": 3, "rule": "DOT001", "message": "Deadlock risk: Non-terminal state 'OrphanNode' has out-degree 0.", "reflex": "Add at least one outgoing transition from 'OrphanNode' or rename it as a terminal sink." }, { "line": 6, "rule": "DOT001", "message": "Deadlock risk: Non-terminal state 'Processing' has out-degree 0.", "reflex": "Add at least one outgoing transition from 'Processing' or rename it as a terminal sink." }, { "line": 4, "rule": "SCAR-PROC-152", "message": "Async state 'AwaitingLLM' lacks explicit teardown/cancel/error exit edge (out-degree=1).", "reflex": "Add an explicit transition from 'AwaitingLLM' to a Disposed, Cancelled, or Error state." },