Numbers we’ll stand behind, losses included.
18 benchmarks against real production codebases: blind adjudication by an independent judge, the same corpus for both tools, and a separate audit against detector tampering. 12 are wins, 3 are losses today, and 3 have no competitor to measure against yet. More than once a number we had already published went stale and was replaced with the worse, verified one. All of it is in the table, published as measured.
An independent judge model rules each finding correct or wrong, not self-reported by either tool.
One real, shared codebase for both tools per language, no cherry-picked test fixtures.
A documented audit specifically checks that no detector was hollowed out or removed to flatter the number.
18 benchmarks. 12 wins, 3 losses.
Wrong-match rate is the number that matters: of everything a tool reports, how much of it is factually wrong. Lower is better. Every row below went through independent blind adjudication, and every row names the real codebase it was measured on. Rows we lose are sorted into the same table rather than kept off it, and rows where the competitor was not re-run against the fresh pool say so on the row.
Most of these started as a loss.
We publish the arc, not the endpoint. A precision number you can only see at its best tells you nothing about whether it will hold. One of these arcs is a correction to a number we had already shipped, and three have not turned yet: Django, Bash and SCSS are losses today, sitting in the table above rather than waiting offstage for a better round.
A confirmed loss to actionlint on its home turf, fixed by file-type gates that silenced 496 false positives and reached 11.6%. A fresh round-2 pool moved it back up to 19.5%; the fixes held, and the higher number is the one published.
A correctly-scoped run found a real regression: the detector surface had moved and nobody re-checked it. Eight structural defects, then ten more mined from the judge’s own reasoning on a tie.
Behind on the first blind panel, still tied after round 2, and only won on round 3 by fixing category by category instead of applying one blanket rule.
One meta-pattern explained 35 of 41 wrong categories: detectors matching a keyword shape without confirming the AST construct they claimed to flag. 142 regression tests.
Detectors carrying a premise that does not hold for a resilience library, then a regression caught by a same-repo reverify: four detectors shipped without the type gate their own neighbors already proved necessary. Fixed same day, recomputed to 9.1%.
Three rounds against a real production app, plus one caught regression: an unrelated engine fix silently tripled the finding pool under an already-merged win. The re-collect exposed two real bugs, Vue files getting JSX rules and an unanchored console.log check, both fixed before the number was re-proved.
A published, completely-judged loss to ESLint+react. Nine detectors checking whole-construct properties a single-line regex cannot scope were quarantined with a recorded reason each, after a constructed recall test proved no window size was recall-safe. A shared loader bug letting quarantined detectors keep firing was found mid-round and fixed.
Seven rounds, two flips, one measurement artifact: a flat ±35-line excerpt was making judges mark correct whole-function findings wrong, fixed in the harness for every future campaign. Six shared-detector bugs, three quarantines, and one judged-wrong finding left unchanged because SQLAlchemy’s own docs say it is correct.
The generalization check the 0.0% result was missing: a second Gin corpus broke the surface at 40.4% on first contact. Three fixes, five quarantines, and one engine bug underneath them all: the per-function scan path computed match positions from the wrong baseline, so every negative-context check on that path silently suppressed nothing. One detector went from 66 raw findings to 1 the moment it landed, with no change to its own rule.
On FastAPI we now beat Semgrep on wrong-match, 10.9% against 17.1%, same corpus, same judge, and 10.9% is at our own <10% bar rather than under it: we are not rounding it down. Real-rate is still the open gap, 27.7% against a 70% target, and Semgrep is not clean on it either at 39.0%. The round-5-to-6 improvement includes a judge change alongside the fixes, disclosed in the write-up; what is not judge-dependent is the raw count, 391 false positives removed from the same corpus across rounds 6 and 7.
On GitHub Actions we lead only against actionlint’s round-1 number, our own 19.5% is above our 10% bar, and actionlint still finds more genuinely worth-fixing correctness bugs than we do: its expression and action rules catch invalid property references and deprecated versions that need a full workflow-expression AST. The same gap shape appears in C#, where two XML-doc cases need signature-to-doc-order correlation a line scan cannot do. All of it is disclosed rather than folded into the headline.
Four comparisons carry a competitor number that was not re-run against the same fresh pool: Java, GitHub Actions, Bash and Go show the rival’s round-1 figure next to a fresh TrueMend measurement. Each of those rows says so. Go is the sharpest case: our own side moved from 40.4% to 5.9% while golangci-lint’s number stayed where round 1 left it, so the row is a win against a figure nobody has re-measured. And several of our own numbers sit above the 10% bar we hold ourselves to even where we lead, Java and GitHub Actions among them.
Express’s 9.9% is judge-dependent, and we say so. Round 3’s first pass used a stricter judge model and read the identical 596-finding sample at 23.3% wrong; running 407 findings through both judges showed they draw the line between “literally present” and “worth fixing” differently, agreeing 85% of the time. The certified number uses the same judge as every earlier round, so it is directly comparable, and the stricter reading stays on record. React’s round-3 win carries the same caveat in the other direction: rounds 2 and 3 used different judges, so only the same-judge round-3 gap is claimed, its 22.1% real-rate is still far under the 70% bar, and a same-judge re-measure has not been run yet.
Detectors get switched off rather than narrowed when no honest version of the rule survives scrutiny. A Promise.all fail-fast check went, after a same-file minimal pair proved the difference was architectural judgment with no regex-visible signal. So did a C# null-check whose premise was wrong for type patterns, a design-pattern heuristic that was a matter of taste, a production-logging rule that misread intentional framework logging, and four GitHub Actions checks whose premise cannot be verified from a trigger event at all.
One went the other way. cq-deeply-nested-callbacks was judged wrong on every sampled instance and we kept it anyway, because disabling it would have cost real callback-pyramid recall. That is a trade-off, not a fix, and it stays disclosed as one.
Nine are still unmeasured.
Twenty-two languages ship. Eleven have a head-to-head number against that language’s default linter or scanner, and two more, Kotlin and Swift, carry a first framework baseline with no competitor wired in yet. The rest go through the same blind-judge, same-codebase, tamper-audited process before they appear here, not before.