There's a stat going around this year about vibe-coded apps needing rescue engineering, six figures a project, a checkbox nobody checked before the thing had real users. It's a real problem. But the part that actually matters isn't the horror story, it's the question underneath it: how do you know a tool that claims to catch this stuff is telling you the truth?
Most benchmarks answer that by picking the number that flatters them. We built ours to do the opposite. Every TrueMend finding in the published competitive benchmark gets checked against the actual source before it's allowed into the document, and the losses get reported next to the wins, in the same table.
Look at Bash. 18.5% wrong-match versus ShellCheck's 19.0%. That's barely a win, and it used to be worse, a prior round measured 33.2% before a real fix pass brought it down. SCSS is an outright loss in one round, stylelint's checks are almost entirely syntactic so it has almost no room to be wrong, and it took a full round of quarantining context-blind detectors to even match it at 0%. We published both numbers. Neither one is the headline you'd pick if the point was to look good.
A benchmark that only wins is worthless. The first pass of our own FastAPI run flagged four critical "auth bypasses" that were all false positives, on our own product, before a single number went in the document.
What actually happened, mechanically
The FastAPI false positives are a good example of what "deterministic-first" catches when you turn it on yourself. Here's roughly the shape of the bug:
// what the detector flagged as critical
// semgrep-auth-bypass-default matched on the literal word "password"
self.password = password; // a variable assignment, not a secret
function login({ password }) {} // a destructured kwarg, not a secret
// and once, a docstring that happened to contain the word
// the fix
// require a quoted literal before flagging a hardcoded credential
const HARDCODED_SECRET = /["']([a-zA-Z0-9+/=]{16,})["']/;
// a variable name or kwarg no longer matches; only an actual literal does
That's one of fifteen detector defects the benchmark surfaced and fixed before publishing, across two languages, all documented with the before/after and the commit. Some of them are almost funny in retrospect: an unused-import checker that flagged every single "import os" as dead code, a regex so broken it fired on its own example in its own docstring. All fifteen shipped with regression tests.
The point isn't that these bugs existed. Every static analysis tool has false positives, that's not news. The point is what you do when your own benchmark catches your own tool lying to you. You fix it and you show your work, or you quietly drop the run and try again until the number looks better. We did the first one, on the record, with the commit hashes.
Where it still loses, plainly
Ruff is roughly 33 times faster than TrueMend on the same codebase. Pylint's data-flow analysis catches a class of correctness bugs TrueMend's Python path is thinner on. CodeQL does interprocedural taint analysis, tracing untrusted input from source to sink across functions and files, and TrueMend doesn't do that at all, categorically, no hedging. None of these are close calls we're spinning. They're just true, and a tool that hides its losses is a tool you can't trust on its wins either.