Methodology · measured 2026-07-19

We run the engine on itself, and publish what it says.

Every number on this page came from a real run against this repository’s own working tree, not an estimate and not a demo corpus. The most uncomfortable one is the false-positive rate, so it gets the most room.

Start here

Two precision numbers exist. They are not interchangeable.

Conflating them is the single most common way a static-analysis vendor overstates itself. One says a rule does not contradict its own documentation. The other says the rule is right about your code. Only the second one is worth anything to you, and it is the harder, slower, more expensive measurement.

SELF-EXAMPLE CONSISTENCY · CI-ENFORCED
99.6%
4,732 / 4,752 patterns pass

Every pattern that ships a good and a bad example must fire on its own bad example and stay silent on its own good one. This runs on every spec-editing pull request, so it is always current. It is also narrow: it proves a rule is internally consistent, not that it is accurate against real code.

Twenty patterns fail, all in the same direction: the rule does not fire on the bad example it ships. They sit in a handful of newer specs, named below.

REAL-CORPUS ADJUDICATION · SLOW
−27%
failing 1,854 → 1,362 over ten rounds

A ten-round campaign judging findings against real source rather than bundled examples. Passing went 3,004 → 3,843 across the same rounds. This is the number that reflects what a rule does in the wild.

It is also the number we cannot refresh cheaply. Last updated 2026-06-01 and not re-run since: a full re-run is a multi-round campaign, not a build step. Treat it as a snapshot with a date on it.

Dogfood

What the engine says about the engine.

Raw finding counts, not adjudicated rates. Both targets are down materially against the baseline the refactor plan cited, and the test tree scores far worse than the source tree, which is the honest shape of a codebase where test fixtures deliberately contain things that look like defects.

src/317 files analyzed
HEALTH SCORE
92.84
FINDINGS
3,064
VS BASELINE
−32%

2 critical, 115 high, 2,172 medium, 38 low, 737 info. The plan cited 4,490 findings as the baseline for this tree.

tests/690 files analyzed
HEALTH SCORE
66.17
FINDINGS
13,587
VS BASELINE
−67%

0 critical, high or medium. 2,405 low and 11,177 info. Against a cited baseline of 40,911, and the low score is expected: fixture code contains defect-shaped strings on purpose.

The number most vendors do not print

Most of what we report on our own source is wrong.

A stratified random sample, seed 42, capped per detector id so no single noisy rule dominates the draw. Every sampled finding was read against its actual source at file and line, and judged skeptically in both directions: a finding was not marked correct because the rule sounded reasonable.

SRC/ · 48 DISTINCT DETECTOR IDS
58.8%overall wrong
sample 102 of 3,063 findings
critical100.0% · n=2 · census
high73.3% · n=30
medium50.0% · n=40
low33.3% · n=15
info73.3% · n=15
TESTS/ · 84 DISTINCT DETECTOR IDS
38%≈34.8% population-weighted
sample 100 of 13,587 findings
low57.1% · n=35
info30.0% · n=60
warning / error0.0% · n=5 · census
READ THIS BEFORE THE HEADLINE NUMBER

Both critical-severity findings in the entire 3,063-finding source run were false positives. One flagged a compile() call used purely for syntax validation, the other misread re2.compile() as the dangerous Python builtin. A two-finding census is not a rate, but it is exactly the kind of result that gets quietly dropped, so it is printed here instead.

Named root causes

The false-positive mass concentrates in identifiable, fixable causes rather than spreading as random noise. That is the difference between a detector surface with a bug list and one that is unreliable in ways nobody can enumerate.

cq-error-return-noneIgnored the exemption for Optional returns that its own spec documents.4 / 4 wrong
semantic-except-no-loggingDid not recognize the deliberate except BrokenPipeError: pass idiom.4 / 4 wrong
semantic-json-load-unboundedFired on small local config files regardless of their actual size.4 / 4 wrong
two @staticmethod detectorsThe regex never inspected the method signature it was reasoning about.5 / 5 wrong
semantic-high-swallow-ratioDid not recognize this codebase’s own documented noqa BLE001 convention.4 / 5 wrong
secret / SQL-injection rulesOn tests/, the largest single bucket: literal strings that exist only as input fixtures for another detector’s regression test.8 / 38 wrong
Speed

Measured on more files than last time, in less wall clock.

The file count analyzed grew 72% while total time dropped, so this is a real improvement rather than an apples-to-oranges comparison against a smaller corpus.

13.8 s
FULL SRC/ · 317 FILES · --CI

Parallel run across the whole source tree. The prior measurement covered 184 files in 17.0 s.

0.34 s
SINGLE FILE · CACHE-COLD

First analysis of one file with nothing warm, the worst realistic single-file case.

0.26 s
SINGLE FILE · CACHED

Re-run of the same file, which is what an editor or pre-commit hook actually pays.

test suite: 10,777 collected · 10,728 passing · 49 skipped · no source file exceeds 1,500 LOC, verified by full sweep rather than sampling
Not current

What on this page has a stale date.

A measurement without a date is a claim. These are the ones we know are behind, listed so nobody has to discover them by being wrong in a meeting.

2026-06-01

The real-corpus adjudication campaign and its 1,362-failing figure. Not re-run since, and it predates the refactor work described above. A full re-run is a multi-round campaign rather than a build step.

not re-profiled

The remaining-bottleneck list from the April speed work: regex call counts, keyword-gate tuning and the rest. The 26x optimization history behind them is accurate as history, and the open items should be treated as stale until profiled again.

open

Twenty patterns fail their own bundled bad example, in Azure Functions, Blazor, EF Core, GitHub Actions, Kubernetes security, Next.js UX, Svelte UX, Terraform, Vue UX and YAML. Each needs either a pattern fix or an example fix.

Check the arithmetic yourself.

The head-to-head benchmarks against each language’s default linter or scanner use a separate process: blind adjudication by an independent judge, on a shared real codebase.

· hello@truemend.ai · code, made true