Skip to content
agentprint

Benchmark

A public, repeatable test of how well agentprint finds keys in the files AI coding agents write, next to two well-known scanners. The corpus is synthetic and the labels are ours, so read it as a regression check you can rerun, not as a ranking.

Run of 2026-11-07. Corpus: 195 cases, 118 with one fake key and 77 with look-alikes only. Tools: agentprint 0.1.1, gitleaks 8.30.1, trufflehog 3.99.8. All three tools were included in this run.

Overall

Section titled “Overall”
Tool Version Recall Recall, plain keys Recall, hidden keys Precision Keys found False positives Recall, any hit in file
agentprint 0.1.1 100% 100% (69 of 69) 100% (49 of 49) 94% 118 of 118 7 of 77 100%
gitleaks 8.30.1 53% 78% (54 of 69) 18% (9 of 49) 82% 63 of 118 14 of 77 53%
trufflehog 3.99.8 53% 77% (53 of 69) 20% (10 of 49) 100% 63 of 118 0 of 77 53%

Plain keys are the rows where the key is written out as an agent writes it; hidden keys are the rows where it is hidden by one trick. 49 of the 118 keys are hidden, so the overall recall leans on them. The hidden-key rows favour agentprint: they test the tricks this release (agentprint 0.1.1) added, and were written alongside the new detection. Compare the tools on plain keys first.

How a case is scored, the same for every tool. Each fake key sits on known lines of its file (the manifest records the span when the corpus is built). A key counts as found only if a finding’s reported line falls inside that span. A finding on the key’s line counts even when only part of the key matched: some of agentprint’s “hidden: split string” hits, for example, are matches of the key’s first part, not the whole joined key. Recall is found / (found + missed). A negative case (look-alikes, no key) counts as a false positive if the tool reports anything in its file. Precision is found / (found + false positives); it is file-level, not per-finding. The last column is the looser reading, any finding anywhere in a key’s file, shown so the two can be compared. Each case file holds one key, so the two differ only where a tool reported something else in the file but not on the key; in practice that is the paired AWS secret access key, which agentprint’s generic rule flags on its own line.

Keys the tool should find

Section titled “Keys the tool should find”

Plain rows are keys in the form an agent writes them (a history line, an allowlist entry, an MCP env block). The other rows hide the same kind of key in one trick. Each row holds seven cases, one per key format: Anthropic, OpenAI, GitHub, AWS, Stripe, Slack, Google. Every AWS case also holds a fake secret access key (40 characters, FAKE first) on a different line, because some scanners, trufflehog among them, only report an AWS key when the pair is present. Only the key ID’s line counts as the key’s span, so a tool that finds just the secret scores a miss, and a hidden key is never rescued by the visible secret.

Key written as Cases agentprint gitleaks trufflehog
plain, in ai providers 6 6 of 6 2 of 6 5 of 6
plain, in claude history 7 7 of 7 6 of 7 6 of 7
plain, in claude mcp 7 7 of 7 6 of 7 6 of 7
plain, in claude settings 7 7 of 7 6 of 7 6 of 7
plain, in codex toml 7 7 of 7 6 of 7 6 of 7
plain, in context only 4 4 of 4 2 of 4 2 of 4
plain, in cursor mcp 7 7 of 7 7 of 7 6 of 7
plain, in dev platforms 13 13 of 13 10 of 13 7 of 13
plain, in hermes yaml 7 7 of 7 6 of 7 6 of 7
plain, in publishing 4 4 of 4 3 of 4 3 of 4
hidden: base64 7 7 of 7 6 of 7 6 of 7
hidden: line continuation 7 7 of 7 1 of 7 2 of 7
hidden: quotes side by side 7 7 of 7 0 of 7 0 of 7
hidden: split concat 7 7 of 7 1 of 7 1 of 7
hidden: split string 7 7 of 7 0 of 7 0 of 7
hidden: yaml folded 7 7 of 7 0 of 7 0 of 7
hidden: zero width 7 7 of 7 1 of 7 1 of 7

Look-alikes the tool should leave alone

Section titled “Look-alikes the tool should leave alone”

Each of these is a file in an agent shape holding text that looks random but is not a key. Lower is better.

Look-alike Cases agentprint gitleaks trufflehog
base64 image 7 0 flagged 0 flagged 0 flagged
empty quotes 7 0 flagged 0 flagged 0 flagged
git sha 7 0 flagged 0 flagged 0 flagged
jwt in token field 7 7 flagged 7 flagged 0 flagged
jwt shaped 7 0 flagged 7 flagged 0 flagged
lorem ipsum 7 0 flagged 0 flagged 0 flagged
minified js 7 0 flagged 0 flagged 0 flagged
placeholder 7 0 flagged 0 flagged 0 flagged
sha256 7 0 flagged 0 flagged 0 flagged
short base64 7 0 flagged 0 flagged 0 flagged
uuid 7 0 flagged 0 flagged 0 flagged

Method

Section titled “Method”
  • The corpus is built at run time by bench/build_corpus.py with a fixed seed (20261107), so every run sees the same files. Nothing key-shaped is committed to the repository.
  • Keys use the real format of each provider (prefix, alphabet, length). Bodies are random characters that start with the marker FAKE, so they are not credentials. They are never sent anywhere or tested.
  • Every case is one folder with one agent file at the path that agent uses: Claude Code history.jsonl, settings.local.json and .mcp.json, Codex config.toml, Cursor mcp.json, Hermes config.yaml.
  • agentprint runs through its public command line, once per case, and the numbers describe whatever version is installed: agentprint scan --home <case> --format json --fail-on none --plain <case>. Only its secret checks are counted (AP001 history, AP002 settings, AP005 MCP config); permission, hook and file-mode findings are not key detection and are ignored.
  • gitleaks runs once over the whole corpus: gitleaks dir <corpus> --no-banner --redact --exit-code 0 --report-format json --report-path <file>. trufflehog runs once too: trufflehog filesystem <corpus> --json --no-verification --no-update. Both use default rules, offline. Their versions are pinned, with checksums, in the Benchmark workflow (.github/workflows/benchmark.yml); the version used is in the table above.
  • Before results are written, each tool must have found at least one plain key on its line, and at least one of those must sit in a hidden agent folder (.claude, .codex, .cursor, .hermes), which shows it really read them; otherwise the run fails instead of publishing a broken number.
  • Only counts, case ids and versions are stored. No key value, matched text or finding detail is printed or saved.

Limits

Section titled “Limits”
  • The corpus is synthetic. Real history files are messier, and real keys are rarer than in this corpus.
  • The labels are ours. We chose the cases, the shapes and which tricks count as hidden keys.
  • Tools run with default rules and no tuning. gitleaks and trufflehog are general scanners and agentprint knows agent file layouts; a different configuration would change their numbers.
  • trufflehog is run without verification, so it never contacts a service, which is also how it is least able to rule out look-alikes.
  • Obfuscation rows test tricks a scanner may reasonably not target. A miss there is a known limit, not a bug in the corpus. Secrets assembled by a script at runtime are not in the corpus and cannot be found by reading files.
  • The “jwt in token field” row is a hard case on purpose: a token-shaped value under a secret-sounding name. Flagging it is defensible; it is listed so the cost shows.
  • Results move when tool versions or rules move. Compare runs by version.

The numbers come from bench/results.json; rerun them with uv run python -m tools.benchmark or the Benchmark workflow on GitHub.