Full-pool blind adjudication on NetBox, round 5: TrueMend 25.0% wrong-match vs. a Bandit+Pylint+Ruff bundle's 23.0% weighted average. This is a disclosed loss, not a win — round 4's real fixes held on re-certification but only covered 6 of 100+ wrong categories firing on this corpus.
NetBox is a real production Django app — 1,284 files in its core package. We ran TrueMend against it three separate times, each time with a fresh, independently blind-judged sample, because the first two rounds taught us not to trust a number until it survives re-certification. The honest answer, as of round 5: TrueMend is still worse than the competition here.
Method — blind two-axis adjudication, full pool
Every finding is judged correct or wrong by an independent model that doesn't know which tool produced it. Round 5 drew a fresh, from-scratch sample — 685 TrueMend findings, 445 Ruff, 504 Pylint, 69 Bandit — from the live finding pool, not a re-check of previously-judged sites. A same-day instrument-integrity sweep confirmed zero corrupted verdicts in this round's evidence.
The three-round arc
| Round | Judged | Wrong-match | Note |
|---|---|---|---|
| 1 (baseline) | 963 | 37.1% | Full-pool, first honest number |
| 3 (certified) | 716 | 23.7% | Fresh blind sample — replaced an earlier "still-emitted" proxy that read 9.4% but didn't hold up under re-certification |
| 4 (fix pass) | — | — | 6 categories fixed and generalized, commit ec2ea357 — verified on 5 fresh instances per category, no wrong-match re-measure yet |
| 5 (certified) | 685 | 25.0% | Fresh blind sample — essentially flat vs. round 3 |
Result
TrueMend: 25.0% wrong-match. The Bandit+Pylint+Ruff bundle: 23.0% weighted average. TrueMend is worse than the bundle, and worse than Ruff on its own:
| Tool | Judged | Wrong-match |
|---|---|---|
| Ruff | 445 | 14.4% |
| Pylint | 504 | 28.2% |
| Bandit | 69 | 40.6% |
| Weighted average | 1,018 | 23.0% |
(445×14.4 + 504×28.2 + 69×40.6) / (445+504+69) = 23.0%. TrueMend's 25.0%
sits above that line. This is not a design trade-off and not close enough to
call parity — it's a disclosed gap.
What round 4 actually fixed, and what held
Round 4 (commit ec2ea357) targeted the 6 categories round 3 named as still
broken: django-ux-no-cache (disabled — call frequency isn't verifiable by
line-scan), django-ux-no-pagination (47→10 raw findings), django-ux-no-pagination-class
(11→0), django-ux-no-index-on-filter-field (disabled — db_index isn't
verifiable without model introspection), django-ux-get-or-404-missing
(28→1, and a genuine recall bug fixed along the way), and python-sq-s2091-xss
(47→36, HIGH-severity true positive kept intact).
Round 5's fresh sample confirmed these held: 4 of the 6 categories sampled at
zero wrong (django-ux-no-cache, django-ux-no-pagination-class,
django-ux-no-index-on-filter-field, django-ux-get-or-404-missing). The
other 2 came back wrong — django-ux-no-pagination at 8/8 and
python-sq-s2091-xss at 7/8 — but that's not a broken promise. Round 4's own
report disclosed both as unfixed residuals up front: serializer-instantiation
false positives on pagination, and pure-literal icon-HTML false positives on
the XSS check that were deliberately left firing to protect recall on a
HIGH-severity detector. The fix-then-verify discipline held; the categories
it didn't touch didn't move.
Still open
Wrong-match went from 23.7% to 25.0% — statistically flat, arguably worse —
because round 4 covered 6 of well over 100 distinct wrong categories firing
on this corpus. The top untouched offenders in round 5's sample:
semantic-unguarded-attribute-access (8/8 wrong),
django-ux-no-select-related (7/7), django-ux-model-no-str (7/7),
django-ux-no-atomic-transaction (6/6), and django-ux-no-fixture-data
(5/5). The gap here is breadth, not methodology — round 4 proved the
generalize-and-verify-on-5-fresh-instances process is trustworthy, not
overfit. Closing the remaining ~2 points to parity with the bundle, and the
further distance to beat Ruff outright, needs the same per-category work
applied across the rest of the surface, not another round of high-visibility
fixes.