Round 1 had TrueMend winning at 18.5% vs. ShellCheck's 23.1%. A fresh, much larger full-pool sample in round 2 put TrueMend at 33.2% — worse than ShellCheck's last known number. Five targeted fixes verified as genuinely held; the aggregate regression is disclosed, not hidden.
Round 1 on nvm's Bash scripts had TrueMend ahead: 18.5% wrong-match on 195 judged findings against ShellCheck's 23.1%. That result held up until a fresh, from-scratch full-pool sample came in five times larger. It didn't survive the larger sample, and we're publishing the regression, not the old number.
Round history
| Round | Corpus | TrueMend wrong-match | Judged | ShellCheck wrong-match |
|---|---|---|---|---|
| Round 1 | nvm (smaller pool) | 18.5% | 195 | 23.1% |
| Round 2 (2026-07-28) | nvm (2,240 raw findings, full-pool) | 33.2% | 394 | 23.1% (not re-run) |
The result
Round 2: TrueMend 33.2% wrong-match, 15.0% real-rate, 394 judged. ShellCheck's 23.1% in that table is its round-1 number, not re-measured this round — ShellCheck was not re-run in round 2, so this is not a clean apples-to-apples comparison. Stated plainly anyway: on the fresh, larger sample, TrueMend went from winning (18.5% vs. 23.1%) to worse than ShellCheck's last known number (33.2% vs. 23.1%). That is a real, disclosed regression in the aggregate number, not a rounding artifact.
Five fixes, independently re-verified as holding
The fix pass this round targeted five detector categories and closed a real
root cause: src/core/code_masking.py didn't mask bash multi-line
double-quoted strings, so prose inside heredoc-style strings matched
code-construct regexes. bash-scp-without-verification,
bash-external-vs-builtin, bash-ci-missing-debug-flag,
bash-tmp-clean-up, and bash-source-dot were the five categories fixed
off that root cause, plus two collateral bugs caught during verification
(a single-quote-embedded-double-quote misread, and command-substitution
VAR="$(...)" wrongly blanked as literal string data).
All five were independently re-checked against fresh instances on nvm, not just re-run against the original failing cases — they sample as genuinely eliminated. The regression in the aggregate number isn't a reverted fix. It's the same pattern confirmed independently across Java, SCSS, and Go this same round: the fresh round-2 sample is far larger than round 1's pool (2,240 raw findings vs. the prior smaller pool) and surfaces well over 100 categories the fix pass never touched. Five fixed categories against 100+ untouched ones move the aggregate number by sample composition, not by regression in what was actually fixed.
The code_masking.py fix is a shared engine bug, not a Bash-only patch —
it benefits every language that hits the same masking code path, the same
way a parallel C-style /* */ block-comment masking bug found via Go this
same round benefits every C-style-comment language.
Still open
- ShellCheck needs a fresh round-2 re-measure. 23.1% is a round-1 number carried forward for comparison, not a same-round measurement — this is disclosed above, not buried, but it means round 2 is not yet a true apples-to-apples result.
- 100+ untouched categories surfaced by the larger sample need the same per-category root-cause-and-verify work the five fixed categories got this round. Until that happens, the aggregate wrong-match number stays above the fixed categories' true state.
- 33.2% is above the <10% bar this benchmark measures against. Round 1's 18.5% was already above bar; round 2's number is worse, not closer.