I fact-checked a blockchain's docs. 4 claims fail, 54 can't be checked, and one won't sit still.

I ran one command against the public endpoints of a blockchain called FLOP. In under a minute it had checked 239 claims taken from the project's own documentation. I ran it five times. This is what came back.

PASS 175 to 176 FAIL 4 UNVERIFIABLE 54 to 55 SKIPPED 5 

238 of the 239 claims gave the same answer in all five runs. The same four failed every time. One claim would not sit still. I will explain why that is a finding and not noise.

The command spent 0 FLOP. It is read-only. It is capped at 80 requests. My runs used between 65 and 78 of them. Each run took between 32 and 43 seconds. You can run it yourself.

What this is

FLOP is a devnet where AI agents pay GPU miners for inference. Its protocol promises proofs of the work, a window to challenge sessions and stake slashing for proven fraud.

flop-lab is a set of open tools that put those promises next to the evidence a stranger can actually get. It is a community project. It is not an official FLOP product. The devnet is disposable and can reset. Nothing here moves real money.

Run it

You need Node 22 and pnpm. I tested these steps from a fresh clone.

git clone https://github.com/zkasuran/flop-lab && cd flop-lab pnpm install --frozen-lockfile pnpm --filter @flop-lab/cli... --workspace-concurrency=1 run build node packages/cli/dist/cli.js harness run --profile full --class read-only \ --max-requests 80 --max-flop 0 --out run.json jq -r '[.results[].status] | group_by(.) | map("\(.[0]) \(length)") | join(" ")' run.json 

The exit code means something. 0 means every selected check completed. 1 means a failure was confirmed. 3 means some checks could not finish. That makes it usable in CI.

A check is only marked FAIL after it fails again on a fresh repeat. Each of the four FAILs has two failing attempts behind it, in every run.

160 of the 239 claims quote the protocol's yellow paper. Another 40 come from the tool list the MCP server advertises to agents. 36 come from the public docs.

The number that matters is 54

Every claim ends in one of three states. PASS and FAIL are what you expect. UNVERIFIABLE is the honest third state. It is where the harness earns its keep.

54 claims could not be checked from outside in every one of my runs. They are not necessarily wrong. The public interfaces simply give a reader no way to find out.

• 30 are parameters the spec lists but the public catalog has no row for.
• 15 are runtime constants the public surfaces do not expose. The spec's number cannot be compared with what is deployed.
• 3 are tools that return text with no output schema.
• 6 are other cases. One example is an offer list that was empty, so pagination could not be tested.

I think of each UNVERIFIABLE as a feature request for the public interface. A promise you cannot test is a promise you have to take on trust. Removing the need for trust is the whole point of a blockchain.

The one claim that would not sit still

settlement-conservation checks that across the whole settlement tape, miner pay plus unused escrow plus burn equals the escrow. It passed in three of my five runs. In the other two it came back UNVERIFIABLE with this reason: "Settlement tape changed while paginating; no snapshot token is available."

The tape is live. New settlements arrive while the harness reads page after page, so the pages do not describe one moment. The interface gives no snapshot token that would let a reader pin one. The harness could have guessed. It reported that it could not know.

I count that as the 55th gap in the public interface. It is also why your counts may differ from mine by one. That is why I give ranges above instead of a single tidy number.

The 4 that failed

Three come from the MCP server that FLOP advertises for agents:

• flop_agent_pack_manifest and flop_agent_pack_install accept their advertised minimal input and return a tool error.
• flop_knowledge_get returns a response that violates its own advertised output schema.

The last one was already reported upstream as issue 87. At the time of writing it has no reply. That is not a complaint. Maintainers are busy and this is a devnet. It shows how a finding travels. The harness notices. A human files it. Then it waits.

The fourth is the one nobody expected.

A claim that changed in three days

The run the repository recorded on November 5 had 177 PASS and 3 FAIL. All five of my runs today have 4 FAIL. Exactly one stable claim moved. It went from PASS to FAIL.

The yellow paper says max_active_reservations_base is 4. The public source catalog now says 400. The harness ran the check twice and got the same answer both times.

I do not know which one is right. The catalog may be stale. It may also have been updated ahead of the paper. The harness says as much in its own words: "This is source drift, not a runtime verdict." It cannot tell you what the network enforces. It can tell you that two public documents disagree by a factor of 100. It can also tell you that the two agreed when the repository recorded its first run on November 5.

That is the real case for re-running these checks instead of reading a report once. Documentation drifts. A claim that passed on Thursday can fail by Sunday without anyone doing anything wrong.

Does reporting these things do anything?

14 issues have been filed under my account against the FLOP repositories. Nine were errata in the yellow paper. Eight of those were closed as completed. One was closed as not planned.

One of the fixes was a 10x error in the transaction fee split. The paper said 80 ppt plus 10 ppt where the intent was 80% plus 10%. The maintainers corrected it in the 0.5.0 draft.

The other five issues are open tracker items with little or no response so far.

So a patient outside reader does catch real mistakes. Some of them get fixed. Both of those are true.

How it was built and how it broke itself

flop-lab has 38 tools across 40 packages. It has had 94 pull requests since November 5. 93 were merged. Most of the code was written by AI agents. An AI assistant did most of the reviewing and merging at my direction. I set the rules and I decide what ships.

I want to be plain about that, because the obvious worry is that nobody checked it. A gate stands between every agent and the repository. It refuses a change without a green build, tests, honest wording and a README that says what the thing does not verify. Agents cannot push, open pull requests or deploy. A publisher signs each pull request with a did:key. Every deploy is signed too. You can check that without installing anything:

curl -s https://flop.zkasuran.dev/SIGNATURE.json -o SIGNATURE.json git checkout "$(jq -r .commit SIGNATURE.json)" node scripts/verify-signature.mjs ./SIGNATURE.json . 

That prints "valid": true for every tracked file of the commit the site is running. When I wrote this it covered 2,101 files. Change one byte of one file and it fails with File hash mismatch.

A gate does not make a system correct. The build broke itself three ways while it shipped. Each one is worth knowing about.

1. The signature outgrew its own verifier. The signed manifest lists one hash per file. At 1,847 files the signed file passed 200,000 characters. That was the size limit in the verifier, so the deploy refused to go out. The check did exactly what it was built to do. The limit went to 400,000 with a test that signs a larger manifest.
2. A security header made a page dead. A per-request CSP nonce was added. A set of documentation pages had been prerendered, so their scripts had no nonce. Chrome refused every one. The page loaded and never came alive. A sweep of all 39 routes in a headless browser found it. Now a test fails if any page is prerendered.
3. The fix for that briefly broke production. Rendering those pages per request moved a file read from build time to request time. The deployed function did not include the package manifests. The live docs listed 2 guides instead of 37. Checking the live site right after the deploy caught it. The tests had passed.

I mention the third one on purpose. A green build tells you your code works where you built it. It does not tell you what the platform shipped.

The honest scoreboard

At the time of writing the repository has 0 stars, 0 forks and 0 watchers. It is under three days old.

I am telling you because a launch post that hides its own numbers is not one you should trust. This project is about not doing that.

Try it, then tell me what is wrong

If you do one thing, run the harness and compare your counts with mine. Expect 4 FAIL, 175 or 176 PASS and 54 or 55 UNVERIFIABLE. If you see something else I want to hear about it. Either the network moved or I got something wrong.

The idea carries beyond FLOP. Take the claims your own project makes in its README and docs. How many could a stranger test with a script? How many would come back UNVERIFIABLE? That second number is the one worth knowing.

• Site: https://flop.zkasuran.dev
• Source: https://github.com/zkasuran/flop-lab
• Dated upstream findings: https://flop.zkasuran.dev/docs/findings
• Results as JSON: https://flop.zkasuran.dev/api/v1/status