Two Secret Scanners Found 5 of 7 Exposed Credentials Each, but Missed Different Secrets

Earlier this year, I wrote a secret scanner. Regex rules, a Java backend, a Chrome extension that lights up files on a GitHub page when something looks like a credential. It was a portfolio project and I never used it in anger.

Last week I pointed it at my own GitHub account - all 17 repositories, full history - and ran gitleaks over exactly the same commits as a control. I expected to confirm my scanner was worse. It is worse, by every aggregate measure. That turned out to be the least interesting thing in the results.

There were 7 distinct real secrets in my history. My scanner found 5 of them. Gitleaks found 5 of them. They were not the same 5.

The setup

17 repositories, 0 forks, 2 of them private. 1,049 commits in total, cloned at full depth, every non-merge commit reachable from any ref diffed against its parent.

Gitleaks v8.30.1, run per repository with --log-opts="--all" so it walked the whole history rather than just the tip.

My own scanner could not do that at all, which was the first thing I learned.

My scanner cannot scan history. At all.

I had somehow never noticed this. The detection engine only ever receives files from one place: a live call to the GitHub API that resolves the repository's current default branch and lists its tree.

// the only path into the scanner GET /repos/{owner}/{repo}/git/trees/{branch}?recursive=1 

There is no local-path parameter. There is no ref=<commit> on the tree listing. It can see what is in HEAD right now, and nothing else. It also caps itself at 30 files per scan with a 5-second per-file timeout - sensible throttling for a browser session, fatal for an audit.

Which means that as shipped, my scanner structurally cannot find the thing secret scanners exist to find: the credential someone removed from HEAD and left sitting in commit 400.

To get an apples-to-apples comparison at all, I pulled all 15 of its regex rules, its severity labels, its skip-list and its false-positive filters out of the Java service and into a standalone runner that walks history. Same rules, same logic, different plumbing. Everything below labelled "my rules" is that runner, and the gap is noted where it matters.

Worth saying plainly: 15 regexes, and no entropy check anywhere. That becomes important in a minute.

What was actually in there

Deduplicated by value - compared by hashing locally, never by reading them:

7 distinct secrets, in 11 file locations, across 2 of 17 repositories.

• 5 JWT device tokens, hardcoded in hardware test and demo scripts. Each reused across 1–3 files.
• 2 Postgres passwords, written as literals in docker-compose.yml.

Six of the seven are still in public history as I write this. The oldest has been there 296 days. The JWTs: 283 to 288 days each.

The other 15 repositories were clean. Everything both tools flagged there was noise, which I will come back to.

The split

	My rules	Gitleaks
Postgres passwords (2)	2 found	0 found
JWT tokens (5 distinct)	3 found	5 found
Distinct secrets found	5 of 7	5 of 7
Raw findings, all repos	66	36
Of those, real	7	9
False-positive rate	89%	75%

Two tools. Same commits. Same seven secrets sitting there. Each one found five, each one missed two, and the overlap is three.

Neither of them was sufficient alone. I only know the true number because I ran both.

Why mine missed the JWTs

My scanner's false-positive filter discards any file whose path contains the substring test.

That is a reasonable heuristic. Test directories are full of fake tokens - I know, because my own scanner's test suite is full of fake tokens, and the filter dutifully hid those too.

It also hid four real device tokens, because the scripts I used to drive real hardware are named things like test_live_flight.py and test_telemetry.sh. They are test scripts in the sense of "testing a drone on my desk", not in the sense of "unit test fixture". The filename convention and the actual secret risk disagreed, and my filter only knew about the convention.

Two of the five distinct JWTs exist only in files matching that filter. For those two, my scanner's answer is not "low confidence". It is silence.

Why gitleaks missed the passwords

Both Postgres passwords sit in docker-compose.yml under a plain POSTGRES_PASSWORD: key. Gitleaks reported zero findings for the entire history of one of those repositories.

The likely reason: gitleaks' closest default rule, generic-api-key, is entropy-gated. These passwords are 17–18 characters, alphanumeric, and not especially random-looking. They did not clear the bar.

My rules have no entropy gate at all. A regex that looks for password: followed by a literal will match a boring password, because it has no concept of boring.

I want to flag this one as inference rather than fact. Gitleaks v8.30.1 compiles its default ruleset into the binary instead of shipping an inspectable .toml, so I am reasoning from what it did and did not flag, not from reading its rules. If someone knows the actual rule that should have fired here, I would like to be corrected.

The thesis

Both misses have the same shape.

My scanner was too noisy, so it got a path-exclusion filter. Gitleaks was too noisy, so it got an entropy floor. Both filters work. Both filters are why each tool has a blind spot, and the blind spots don't overlap.

Every false-positive filter is a false-negative generator. You do not get to tune out noise for free. You get to choose which category of real finding you are willing to stop seeing, and most of the time you choose it implicitly, in a one-line heuristic written to make a demo less embarrassing.

The test exclusion in my code is six words long. I wrote it so a screenshot of the extension wouldn't be full of fixture tokens. It cost me two of five credentials.

Where the noise actually came from

Both false-positive rates are terrible, and both are terrible for largely the same stupid reason.

A single commit in one repository checked in an entire Python virtualenv. That one directory accounts for 29 of my scanner's 66 findings and 16 of gitleaks' 36 - vendored library internals from requests, urllib3, numpy, pip, protobuf and friends, flagged over and over.

My scanner's skip-list excludes node_modules, vendor, dist, build and .git. It does not exclude .venv or site-packages. Gitleaks fell into the same hole from the other direction.

Strip that one directory out and both tools look dramatically more credible, which tells you something about how much of a scanner's reported precision is really just its vendored-dependency exclusion list.

The rest of the false positives were the predictable kind: env-var templates like ${POSTGRES_PASSWORD} that reference a secret rather than contain one, validation schemas (password: z.string().min(8)), hashPassword(...) calls, Maven wrapper boilerplate, and my own scanner's test fixtures.

A regex sees password and cannot tell a label from a value. That is the whole problem, and no amount of rule-tuning fixes it - it is an argument for entropy and context, which is exactly the thing that then creates the other blind spot.

What I would change

• Scan history, or don't call it a secret scanner. HEAD-only scanning is a linter. The entire risk model of leaked credentials is "it was committed once and removed later."
• Replace path exclusions with confidence downgrades. A file under test/ should score lower, not vanish. Both of my misses would have been visible at low confidence.
• Run two scanners with different philosophies. Not for redundancy, because a regex tool and an entropy tool fail in opposite directions, and the union is the only number I would trust.
• Exclude vendored dependency trees aggressively, and publish the exclusion list, since it is doing more for the reported precision than the rules are.

Honest limitations

• I cannot prove the exposure windows. GitHub's API exposes a repository's current visibility and its creation date, and nothing about whether it was ever flipped between private and public. Every "days lived in a public repo" figure here is inferred from creation date plus current state.
• Commit author date is not push date. One repository's history was authored locally and pushed in bulk later, so author timestamps overstate exposure. I checked this for the real findings; the caveat is general.
• Real vs. false positive is a judgment call. I classified by reading the surrounding code - whether the captured text started with code syntax or looked like an opaque literal - never by reading the value. A different reviewer could reasonably reclassify one or two.
• Seven secrets across two repositories is a small sample. It supports what those two repositories did. It does not support a general claim about my habits, and it certainly doesn't support one about yours.
• Merge commits were excluded from my runner and included in gitleaks' run. Spot-checking found no merge commit introducing anything its parents didn't already have, but I did not verify that exhaustively across 1,049 commits.

The part I would rather not write

Six credentials are still sitting in public history as of this writing, the oldest for 296 days. I found them with a tool I wrote and then never pointed at myself, in projects I have been linking to from a CV.

The tool was not the problem. I built it, I published it, and I never ran it on the one account I had complete access to. The scanner worked about as well as a scanner of that design can work. The gap was that I treated it as a portfolio artifact rather than as something to use.

If you have written one of these, go run it on yourself before you write about it. That part costs an evening and I can tell you from experience that the results are more interesting than the demo.

---