Benchmark results, wins and losses.
Wrong-match rate of Truemend and other scanners on the same codebase, judged blind. The table includes comparisons where Truemend trails, rounds where its rate rose, and rows where the other tool’s figure is out of date. How the judging works.
Who judged. Truemend commissioned and ran every adjudication on this page. The judges are AI models that Truemend operates. Each rules on a finding without being told which tool produced it. This is vendor-run evidence and has not been audited by a third party.
- 9 fresh head-to-head comparisons
- 8 wins
- 1 loss
- 6 earlier comparisons, not tallied
- 3 baselines, no rival
Wrong-match: the share of a tool’s findings judged factually incorrect.
Lower is better. 11 incorrect findings out of 100 reported is 11%. Real-rate, the share judged worth acting on, is given in the row notes where it was measured. Rows where Truemend trails sit in the same table, and any row where the other tool was not re-measured on the fresh pool says so.
- Blind judge
- rules each finding, not knowing which tool produced it
- Same corpus
- one real codebase and commit for both tools
- Removal audit
- a separate check that no detector was removed only to lower the rate
Fresh head-to-heads
Both tools were measured in the same round, on the same pool. Only these rows are tallied as wins and losses.
338 Truemend and 140 Deno lint findings judged. Worth-fixing rate: 2.4% vs 2.1%, within noise at this sample size.
338 Truemend and 140 Deno lint findings judged.
338 Truemend and 359 Biome findings judged. Worth-fixing rate: 3.3% vs 7.2%. Earlier rounds on this corpus were a near-tie: 16.8% vs 15.4%, then 14.1% vs 13.4%.
338 Truemend and 359 Biome findings judged.
Rates are recomputed on the file scope both tools cover. ESLint's raw rate was 11.3%, and 60% of its wrong findings were in vendored build output that Truemend skips as generated code. Sample covers 31.7% of the finding pool. Earlier round: 13.6% vs 8.2%.
Rates are recomputed on the file scope both tools cover.
Same codebase and commit for both tools. A separate audit checked that no detector was removed only to lower the rate.
Same codebase and commit for both tools.
The Clippy sample is small (11 findings judged), so its rate has high variance. 5 of 104 Truemend findings and 2 of 11 Clippy findings were judged wrong. Earlier round: 9.3% vs 9.1% (118 and 11 judged).
The Clippy sample is small (11 findings judged), so its rate has high variance.
A second, stricter judge model rated the same findings at 23.3%, and the two judges agree on about 85% of items. 596 Truemend and 51 Semgrep findings were judged by the same judge model as earlier rounds. One corpus.
A second, stricter judge model rated the same findings at 23.3%, and the two judges agree on about 85% of items.
The judge model changed between rounds 5 and 6, so those rounds are not directly comparable. 976 Truemend and 41 Semgrep findings were judged by one judge model. 10.9% is above the 10% target. Worth-fixing rate: 27.7% vs 39.0%, Semgrep higher.
The judge model changed between rounds 5 and 6, so those rounds are not directly comparable.
Round 2 used a different judge model, and ESLint's rate moved from 8.0% to 19.3% on identical findings across that change, so the rounds are not comparable. 217 Truemend and 88 ESLint + react findings were judged by one judge model in round 3. Worth-fixing rate: 22.1% vs 36.4%, ESLint higher.
Round 2 used a different judge model, and ESLint's rate moved from 8.0% to 19.3% on identical findings across that change, so the rounds are not comparable.
Truemend trails the bundle and Ruff alone. 685 Truemend, 445 Ruff, 504 Pylint and 69 Bandit findings judged. Ruff 14.4%, Pylint 28.2%, Bandit 40.6%; the weighted average of the three is 23.0%. Earlier rounds: 37.1% (963 judged), then 23.7% (716 judged).
685 Truemend, 445 Ruff, 504 Pylint and 69 Bandit findings judged.
Earlier comparisons and recomputations
Either the other tool’s figure is from an earlier round and was not re-run, or the Truemend figure was recomputed on an earlier judged sample. Each row gives the reason and is not counted as a current win.
golangci-lint's 21.4% is from round 1 (59 findings judged) and was not re-measured, so this is not a same-round comparison. The Truemend figure is the latest round (102 findings judged). Round 2 on the same corpus measured Truemend at 40.4% (156 judged).
golangci-lint's 21.4% is from round 1 (59 findings judged) and was not re-measured, so this is not a same-round comparison.
Latest figure is a recompute on the earlier judged sample (77 findings), not a fresh blind run. The fresh round measured 7.7% vs 39.5% (78 and 81 judged). A later same-repository re-check measured Truemend at 12.5% before the fix.
Latest figure is a recompute on the earlier judged sample (77 findings), not a fresh blind run.
The bundle's 60.8% is from round 1 and was not re-measured. Truemend: 485 findings judged, above the 10% target. Real-rate, the share judged worth acting on: 22.9%, up from 5.5% in round 1.
The bundle's 60.8% is from round 1 and was not re-measured.
actionlint's 40.0% is from round 1 (35 findings judged) and was not re-measured. Truemend: 77 findings judged, above the 10% target; round 1 measured 11.6% on a different pool (112 judged). Real-rate: 40.3%. actionlint reported more actionable workflow-expression bugs in round 1.
actionlint's 40.0% is from round 1 (35 findings judged) and was not re-measured.
ShellCheck's 27.3% is from round 2 (194 findings judged) and was not re-run. Truemend: 42 findings judged, above the 10% target. Round 2 measured Truemend at 33.2% (394 judged). Raw findings on this corpus fell from 2,240 to 42 between rounds after rule categories were switched off.
ShellCheck's 27.3% is from round 2 (194 findings judged) and was not re-run.
stylelint's 9.6% is from round 2 and was not re-run. Truemend: 42 findings judged, a small sample with no rival. Round 2 measured Truemend at 45.4% (97 judged). Rule categories were switched off between rounds, which reduces raw findings.
stylelint's 9.6% is from round 2 and was not re-run.
Truemend-only baselines
No comparable tool is in the harness for these. Baselines are published before any fixes, and recomputed figures say what was recomputed.
Recompute, not a fresh sample: a fresh sample of 237 findings measured 19.4%, fixes alone gave 11.6%, and removing one rule category gave 4.5%. That category held 18 findings, 16 wrong and 2 correct, so 4.5% reflects reduced coverage. No ASP.NET-specific rival is in the harness.
Recompute, not a fresh sample: a fresh sample of 237 findings measured 19.4%, fixes alone gave 11.6%, and removing one rule category gave 4.5%.
418 findings judged, drawn from two production Ktor applications (309 files). Real-rate: 14.8%. No Ktor-specific rival tool exists. An earlier result on a library without Ktor code was retired.
418 findings judged, drawn from two production Ktor applications (309 files).
646 findings judged, drawn from IceCubesApp (391 files). Real-rate: 14.2%. SwiftLint is not yet in the harness. An earlier result on a networking library without UI code was retired.
646 findings judged, drawn from IceCubesApp (391 files).
What limits these numbers.
Four limits apply to how far the table can be read, and the rows above state them where they apply.
Rival figures not re-run
Five rows pair a current Truemend figure with the other tool's figure from an earlier round: Java, GitHub Actions, Go, Bash and SCSS. Each row says so, and none is counted in the tally above. In Go, Truemend moved from 40.4% to 5.9% between rounds while golangci-lint's 21.4% is still its round-1 figure.
Our own target
The target is under 10% wrong-match. Rows where Truemend leads the comparison and is still above it: Java 18.6%, GitHub Actions 19.5%, Bash 19.0% and FastAPI 10.9%. Leading a comparison and meeting the target are separate results.
Judge dependence
Express's 9.9% depends on the judge model. A stricter model rated the same findings at 23.3%, and the two judges agree on about 85% of items. The published figure uses the judge from every earlier round, so the rounds are comparable. React has the same dependence in the other direction: ESLint's rate moved from 8.0% to 19.3% on identical findings when the judge changed.
Where the other tool finds more
actionlint reported more actionable workflow-expression and action-version bugs than Truemend in round 1. Real-rate, the share of findings judged worth acting on, is below 50% for Truemend on every row where it is reported, and the other tool's real-rate is also low on most corpora.
Still behind
Python / Django is the row we still lose. On Django (NetBox), the weighted average of Bandit, Pylint and Ruff is 23.0% against Truemend at 25.0%, and Ruff alone is 14.4%.
Reduced coverage
Some rows reached their latest figure partly by switching rule categories off between rounds. That lowers wrong and correct findings together. Affected rows: Go, Bash, SCSS, React and ASP.NET. In the ASP.NET row, removing one category dropped 18 findings, 16 wrong and 2 correct.
How the rate changed between rounds.
Figures from each measurement round, including rounds where the rate rose. A rate shown only at its best does not show whether it holds.
Round 1: 55.7% vs actionlint 44.4%. After the first fixes: 11.6% vs 40.0%. Round 2 on a fresh pool: 19.5% (77 judged).
34.5% on a re-scoped run, 9.3% on the first blind adjudication (118 judged), 4.8% on a fresh blind sample (104 judged).
16.8% vs Biome 15.4% on the first blind panel (279 judged), 14.1% vs 13.4% after round 2, 3.0% vs 8.4% after round 3 (338 and 359 judged).
13.6% vs 8.2% on the first blind pass. On a larger sample: 3.5% vs 11.3%, recomputed to 3.5% vs 5.0% on the shared file scope.
15.6% vs Roslynator 49.4% on the first fresh adjudication, 7.7% vs 39.5% after later rounds, 12.5% on a same-repository re-check, 9.1% recomputed on the earlier sample.
36.9% (872 judged), 11.6%, 17.4% on a fresh sample, 9.98%, then 9.9% (596 judged). A change in the finding pool between rounds 2 and 3 made the earlier figure stale, so round 3 was re-collected.
30.3% (274 judged) vs ESLint + react 8.0% in round 2. Round 3: 10.6% (217 judged) under a different judge model, which rated ESLint at 19.3%.
Rounds 2, 3, 5, 6 and 7 against Semgrep: 24.3% vs 12.2%, 24.4% vs 29.3%, 22.7% vs 26.8%, 12.9% vs 12.2%, 10.9% vs 17.1%. The judge model changed between rounds 5 and 6.
Round 1 on a production Gin API: 0.0% vs Semgrep 50.0% (51 and 40 judged). Round 2 on gin-gonic/gin: 40.4% (156 judged). Round 3: 5.9% (102 judged), Truemend only.
18.5% vs ShellCheck 23.1% in round 1, 33.2% vs 27.3% in round 2 (394 and 194 judged), 19.0% on 42 findings in round 3 (Truemend only).
33.2%, then 11.7% (178 judged, stylelint 0.0%). Round 2 on a fresh sample: 45.4% vs 9.6% (97 judged). Round 3: 0.0% on 42 judged, Truemend only.
Languages without a head-to-head figure.
Detection ships for 26 languages. The languages in the table above carry a head-to-head figure, and Kotlin and Swift carry a first baseline with no comparable tool in the harness. The rest have not been measured this way.
- No head-to-head figure yet
- Ruby, Dart, PHP, C, C++, SQL, PowerShell, Terraform, Dockerfile
- Baseline only, no rival in the harness
- C# / ASP.NET Core, Kotlin / Ktor, Swift / SwiftUI