TrueMend vs. Semgrep on Gin

Gin's framework-specific detector surface hadn't been through real blind adjudication before — only synthetic fixtures. We ran a full campaign against a real production Gin codebase to find out whether that surface holds up outside the lab.

Judged completely, not sampled

The pool was small enough to judge every finding, not a stratified sample — 91 of 91 (51 TrueMend, 40 Semgrep), each ruled correct or wrong by an independent judge with no knowledge of which tool produced it or what the "right" answer should be.

Result

TrueMend: 0.0% wrong-match. Semgrep: 50.0%. Every single one of TrueMend's 51 post-fix findings checked out against the real source. Semgrep's 50% is dominated by one 18-instance false-positive cluster: an HTML-escaping rule firing on Gin's own render/*.go package (json.go, yaml.go, toml.go, protobuf.go, ...) for writing correctly-serialized, correctly-content-typed bytes — the rule doesn't check content type before applying an HTML-context escaping assumption.

What it took to get there

Six systemic detector bugs, all genuine precision fixes — no detector was disabled or withdrawn to reach this number. The largest: an "unchecked error" check that allowlisted specific stdlib comma-ok patterns (strings.Cut, utf8.Decode*) but not Go's own GetXxx() (T, bool) accessor convention, so it fired on every c.GetQuery, c.GetPostForm, and similar call in Gin's own context.go. Fixed by recognizing the GetXxx/Get naming convention directly — two genuine true positives in the same category still fire correctly post-fix. Full root-cause detail for all six: internal engineering notes, available on request.

The follow-up we promised

This case study originally ended here, with one real corpus exhaustively judged, and an open question: would the surface hold up against a second, independent Gin codebase, or did these six fixes just overfit to one repo's idioms? We ran that follow-up. It didn't go the way the first result would predict.

Round 2 — first contact with a second corpus, and it's ugly

The second corpus is gin-gonic/gin's own source tree — its own test suite and internal packages exercise gin.Context, c.Bind/c.ShouldBind, and error handling the same way a consuming application does, just written by the framework's own maintainers with their own idioms. A fresh, from-scratch full pool sample (156 judged, TrueMend-only — golangci-lint wasn't installed this round) came back at 40.4% wrong-match. Confirmed 2 of the 3 prior fixes already held; the number is bad because eight Go/Gin-specific detectors were sitting at 75–100% wrong on idioms this second corpus surfaces that the first one never did — test-helper error-check wrappers (require.NoError), the http.NewRequest-isn't-I/O distinction, and a batch of whole-construct claims (named-return signatures, operand types) no line-scan can see. Publishing a worse number two rounds later isn't a retraction of the 0.0% result above — different corpus, different failure modes — but it's the honest version of "does this generalize," and the honest answer was: not yet.

Round 3 — an engine bug underneath the detector bugs

Eight detectors, each root-caused against the real gin-gonic/gin source. Three were genuine line-scan fixes. Five were quarantined — whole-construct properties (a named-return signature, an operand's type, a return type) that no bounded regex window can see, so we disabled and documented them rather than chase precision with a window tweak that would just trade false positives for silently dropped true positives.

The load-bearing finding wasn't a detector at all. One fix cluster wouldn't narrow no matter how the regex was adjusted, and root-causing it against source turned up an indexing bug in the detection engine itself: the per-function scan path computed each match's position using the wrong baseline, so any detector using a negative-context check on that path silently never suppressed anything — the check ran, found nothing, and kept every match, including the ones it existed to filter out. Confirmed directly against the corpus: one detector's raw findings dropped from 66 to 1 the moment the engine fix landed, with no change to that detector's own rule. It had been correct the whole time; it just never ran correctly.

Round	Judged	Wrong-match	Real-rate	Note
Round 1 (2026-07-26, above)	51	0.0%	—	Production Gin API, vs. Semgrep 50.0%
Round 2 (fresh corpus)	156	40.4%	13.5%	gin-gonic/gin itself — 8 detectors newly exposed
Round 3 (post-fix)	102	5.9%	7.8%	3 fixes, 5 quarantines, 1 engine bug

Under our own 10% bar again, on a corpus that initially broke the surface far worse than round 1's clean result would have predicted. Raw findings on this corpus: 3049 → 2184 (−865 false positives) between rounds 2 and 3. No competitor ran this round — golangci-lint wasn't installed — so this isn't a head-to-head rerun of the 0.0%-vs-50.0% result above, it's the generalization check that result was missing.

Still open

The engine bug's fix applies everywhere the same code path runs, not just to Go/Gin — but this campaign only re-verified it here. Whether other language surfaces were quietly carrying the same silently-disabled negation checks, and how much their own certified numbers move once re-measured, is unverified and the next thing worth checking.