There's a stat going around this year about vibe-coded apps needing rescue engineering, six figures a project, a checkbox nobody checked before the thing had real users. It's a real problem. But the part that matters isn't the horror story, it's the question underneath it: how do you know a tool that claims to catch this stuff is telling you the truth?
Most benchmarks answer that by picking the number that flatters them. We built ours to do the opposite. Every Truemend finding in the published competitive benchmark gets checked against the actual source before it's allowed into the document, and the losses get reported next to the wins, in the same table.
Look at Bash. 18.5% wrong-match versus ShellCheck's 19.0%. That's barely a win, and it used to be worse, a prior round measured 33.2% before a real fix pass brought it down. SCSS is an outright loss in one round, stylelint's checks are almost entirely syntactic so it has almost no room to be wrong, and it took a full round of corrections to match it at 0%. We published both numbers. Neither one is the headline you'd pick if the point was to look good.
A benchmark that only wins is worthless. The first pass of our own FastAPI run flagged four critical "auth bypasses" that were all false positives, on our own product, before a single number went in the document.
What happened
The FastAPI false positives show what a deterministic-first approach catches when you turn it on yourself. The first pass flagged four critical "auth bypasses". All four were false positives, on our own product, and none of them went into the document.
That is one of fifteen detector defects the benchmark surfaced and fixed before publishing, across two languages, each documented with the before and after and the commit. All fifteen shipped with regression tests.
The point isn't that these bugs existed. Every static analysis tool has false positives, that's not news. The point is what you do when your own benchmark catches your own tool lying to you. You fix it and you show your work, or you quietly drop the run and try again until the number looks better. We did the first one, on the record, with the commit hashes.
Where it still loses
Ruff is roughly 33 times faster than Truemend on the same codebase. Pylint's data-flow analysis catches a class of correctness bugs Truemend's Python path is thinner on. CodeQL does interprocedural taint analysis, tracing untrusted input from source to sink across functions and files, and Truemend doesn't do that at all, categorically, no hedging. None of these are close calls we're spinning. They are true, and a tool that hides its losses is a tool you can't trust on its wins either.