TrueMend vs. Clippy on Rust

We don't publish a number until it's been checked twice — once by us, once by an independent judge who doesn't know which tool produced which finding. This is the second check, and it moved the number.

What we found when we checked

Running a fresh, correctly-scoped measurement against tokio's real source (376 files, the same scope our benchmark harness uses) surfaced a real regression: TrueMend's wrong-match rate had drifted to 34.5% against Clippy's steady 14.1% — a clear loss, not a rounding error. An earlier measurement had shown a win; the detector surface had simply moved since then and nobody had re-checked it.

Root cause, not a patch

A wrong-match rate that high is never one bug — it's usually a small number of detectors firing on the wrong evidence, over and over. Direct code reading turned up four structural problems, then four more once we re-sampled after fixing the first batch: a Mutex/Rc deadlock-style check that fired on the mere presence of use std::sync::Mutex without ever confirming the guard was held across an .await; test-file exemption that was filename-only and missed anything under a tests/ subdirectory; a "Box allocation in a tight loop" check firing with no loop in view; a boolean-parameter check flagging every single-bool builder/test-helper argument; and more in the same vein. We fixed all eight with real AST/context checks, added 31 regression tests, full suite green throughout (10,228 passed, 0 failed).

The self-measured number, and why we didn't stop there

That work took the self-measured wrong-match rate from 34.5% down to ~2.1% — which would have made a great headline. We didn't publish it, because "self-measured" means one of us read the sample and judged it, which is exactly the kind of self-grading that makes a benchmark untrustworthy. So we ran it again, blind.

The confirmed result

An independent judge (Cursor, via our own two-axis blind-adjudication harness — the judge sees a code snippet and a claim, not which tool made it or what we hoped the answer would be) rated 118 TrueMend findings and 11 Clippy findings. Result: TrueMend 9.3% wrong-match, Clippy 9.1%. Close enough to call a tie. The self-measured 2.1% was real progress, but optimistic — a single reader is worse at catching subtle wrong-matches than an adversarial second opinion, which is exactly why we built the blind harness in the first place.

One more honest data point: on tokio's internals — a mature, heavily reviewed codebase — the judge found essentially nothing from either tool that a reviewer would actually act on. Even correct, most of what's left to find in code this mature isn't worth fixing. That's a real signal about what "coverage" means once you're past the obvious bugs, not a knock on either tool.

We mined the tie for the next round of fixes

A tie isn't a finish line, so we went back to the 118 judged findings and read every one of the 11 the judge called factually wrong. They spread across 10 distinct detectors, not one bad rule: a numeric-literal check that misread a GitHub issue number inside a debug_assert! message that wrapped across three lines (the per-line string stripper had never learned to follow a Rust string past a line break); a boolean-parameter check flagging compare_exchange(false, true, ...) — the standard, fixed-arity atomic compare-and-swap API, not an ambiguous caller-chosen pair; a TODO/FIXME check that couldn't see a tracking-issue URL sitting a few lines below the comment instead of on the same line; a dead-code check with no way to tell whether the attributed function was actually called elsewhere in the file; an .expect()-in-production check blind to Rust's idiomatic in-file #[cfg(test)] mod tests { ... } convention (test code that lives in the same file, not a separate directory); and more in the same shape — each one a real context-blindness defect, not a matter of taste.

One of those fixes required correcting our own prior work: an earlier same-day pass had written a regression test asserting the opposite of what the fresh judge now confirmed — that compare_exchange calls were "genuinely ambiguous." They weren't. We fixed the detector and the test.

Verification wasn't just re-running our own suite. We re-analyzed the exact real tokio-rust files and confirmed all 11 originally-wrong findings are now silent, then diffed category counts across the whole corpus before and after the fix — not just the judged sample — specifically to catch over-suppression. That diff caught a real bug in our own fix: the dead-code usage check first shipped matching any occurrence of a function's name, including an unrelated local variable that happened to share it. Tightened to require call-shaped syntax, re-verified, corpus counts confirmed the fix without over-suppressing.

Result — re-confirmed, independently, a second time

We didn't stop at "confirmed silent on real source." We ran the whole blind adjudication process again, fresh: a new sample (115 findings, 36% of the pool, comparable coverage to the tied measurement), judged by independent reviewers with no visibility into which tool produced which finding or what we hoped the answer would be.

TrueMend: 4.8% wrong-match (5 of 104 judged). Clippy: 18.2% (2 of 11). A real win — not a rounding-error flip of the tie. Clippy's number moved against it too on this sample; with only 11 Clippy findings in the pool, that's normal small-sample variance, not a claim that Clippy got worse. The honest comparison is TrueMend's own arc: 9.3% → 4.8%, backed by 10 disclosed fixes and two independent judges who never saw each other's verdicts.

Five TrueMend findings are still judged wrong on this sample — three are the same detector (rust_unused_import) where the judge's own note says the source excerpt window didn't show enough of the file to confirm the import really was unused, which may be a sampling artifact rather than a detector bug worth chasing yet. The other two are real, disclosed leads for the next round: a loop-mutation check firing on an idiomatic range for loop with no manual index, and a raw-slice-safety check missing a SAFETY-comment-backed length invariant. We're not hiding those to make the number look cleaner — they're the next thing to fix.