TrueMend vs. Semgrep on FastAPI

We don't only publish the wins. Through round 2, apples-to-apples, Semgrep came out ahead on this one — and we think showing that plainly is more useful to you than a headline number that hides it. Rounds 3-7 below update the picture; read on for the current numbers.

The full-pool result (through round 2)

Judged against Netflix's open-source Dispatch app (a real, sizeable FastAPI production codebase), on the whole finding pool: TrueMend 24.3% wrong-match, Semgrep 12.2%. Semgrep won this comparison. We didn't frame that as a "different philosophy" — it was a real gap, and rounds 3-7 below show it wasn't closed by fixing FastAPI-specific detectors alone.

Why the aggregate number is worse than the FastAPI surface itself

The 24.3% figure is dominated by detectors that have nothing to do with FastAPI — SQLAlchemy patterns, Docker checks, generic TypeScript/JSX rules picked up from the app's frontend assets. This campaign targeted TrueMend's FastAPI-specific detectors, not that whole surface. Scoped to just the fastapi-* categories, after two rounds of fixes: 5.1% wrong-match — a result that would clear our own 10% bar on its own, and one Semgrep has no directly comparable scoped number to set against, since it doesn't ship a FastAPI-specific ruleset.

What got fixed, and what didn't

Round 1 measured a version of the detector surface that was structurally blind to 97% of the app's actual routes (@app.*-only patterns against a codebase that's 1 @app.* and 277 @router.*) — round 2 widened that first, then fixed six real precision bugs: a missing async def gate that made fastapi-ux-sync-orm-in-async fire on ordinary synchronous services (472 false positives, now zero), a broken negative-lookahead regex in the response-model check that only evaluated once instead of per character (164 false positives collapsed to 8), and four more of similar shape. Two detectors were withdrawn rather than patched further, disclosed as coverage cuts, not precision wins: fastapi-ux-no-pagination (the corpus has zero .all() calls in view files once scoped to real route context — a real gap between the detector's claim and this app's shape, not a bug we could fix with a regex) and fastapi-ux-sync-def-handler (correct but never worth fixing on a codebase that's intentionally sync end-to-end).

Rounds 3–5 — a fresh certification flips the lead, then a real fix holds

The full-pool numbers above (24.3% TrueMend vs. 12.2% Semgrep) didn't hold either. Round 3 (2026-07-27) started with an interim "still-emitted proxy" measurement that reported 9.4% wrong-match after a fix pass — a number that turned out to be unreliable by construction: the proxy only re-checks findings at file/line/category triples a prior sample already judged, so it's blind to any fresh instance the fix pass didn't happen to touch. A fresh, from-scratch blind sample (1065 TrueMend findings, judged blind alongside a newly-drawn Semgrep sample — so the round-2 12.2% Semgrep number above isn't like-for-like with what follows) certified the real number instead, and it flipped which tool comes out ahead: TrueMend 24.4% wrong-match, Semgrep 29.3%. The wrong findings were dominated by five detectors that have nothing to do with FastAPI routes: sqla_no_polymorphic (8/8 wrong), ai_config_model_in_code (8/8), python-mutable-class-var (8/8), sqla_cascade_delete_orphan and semantic-orm-writes-no-transaction (8/8 each).

Round 4 (commit ec2ea357) fixed six of them: sqla_no_polymorphic (49→0, demoted to catalog_only — the pattern needs cross-class inheritance resolution a line-scan can't do), ai_config_model_in_code (11→0, test/mock file exclusion), python-mutable-class-var (570→417 combined with Django's share, Pydantic BaseModel field-declaration skip), sqla_cascade_delete_orphan (33→8, scalar-only uselist=False per SQLAlchemy's own cascade docs), python-sq-s2275-assert-message (23→0, the pattern was matching the correct idiom instead of the actual bug shape), and orm-writes-no-transaction (recognizes .commit()/.rollback() as a valid transaction boundary, not just .begin()).

Round 5 certified the fixes held on a fresh full-pool sample:

Round	TrueMend judged	TrueMend wrong-match	Semgrep judged	Semgrep wrong-match
Round 2 (interim, full pool)	1292	24.3%	41	12.2%
Round 3 (fresh, from-scratch)	1065	24.4%	41	29.3%
Round 5 (post round-4 fixes)	1030	22.7%	41	26.8%

sqla_cascade_delete_orphan improved cleanly (8/8 wrong on the round-3 sample → 2/8 on the round-5 fresh sample). python-mutable-class-var improved only partially (8/8 → 4/8): the Pydantic BaseModel check doesn't resolve indirect project base classes like DispatchBase/CaseBase that themselves inherit BaseModel one level removed — exactly the failure mode round 4's own report flagged as possible. New top-wrong categories round 4 never touched: docker-version-arg-not-used, semantic-python-too-many-locals, slop-formulaic-error-handling.

Round 5, certified on a fresh full-pool sample (1030 TrueMend findings, 41 Semgrep findings, judged blind): TrueMend 22.7% wrong-match, Semgrep 26.8%. TrueMend led on that axis. On the other axis it didn't: Semgrep 34.1% worth-fixing/real-rate vs. TrueMend's 22.0% — a higher share of what Semgrep flagged was worth acting on. One axis each. Not yet a clean win — rounds 6 and 7 below are what closed it.

Round 6 — a measurement artifact, and six real bugs

Two independent root causes drove the round-5 number, and round 6 fixed both. First, a measurement artifact, not a detector defect: the adjudication harness cut a flat line ± 35 excerpt, which can't show a judge the 100–150-line function body that whole-function-claim categories (too-many-locals/statements/branches, cyclomatic complexity) actually assert against — so a judge marked genuinely correct findings "wrong" purely for excerpt reasons. Verified on signal/service.py::update (lines 472–621): it genuinely has 31 locals / 91 statements (ast-verified), a correct finding judged wrong. Fixed with a shared function-aware adjudication window that expands to the full enclosing Python function for structural categories — a harness fix, not a detector change, so it carries no recall loss and raises adjudication fidelity for every future campaign.

Second, six real line-scan precision bugs in shared (non-FastAPI) detectors, each root-caused against Dispatch source and recall-verified: ts-any-cast (5/5 wrong → 0, was parsing raw .vue/.svelte SFC content as TypeScript), docker-version-arg-not-used (7/7 → 0, framework build-tool env prefixes like VITE_/NEXT_PUBLIC_ are consumed implicitly and shouldn't require an explicit $-dereference), python-mutable-class-var (4/8 → 0 on this corpus, dunder config attrs and factory_boy declarative fields excluded), exception-no-chain (6/8 → 0, bare re-raises chain automatically via __context__), unguarded-attribute-access (6/8 → 0, the Google API client builder .get(...).execute() idiom excluded), and bash-printf-echo-posix (3/8 → 0, bash-shebang scripts aren't targeting POSIX portability).

Tool	Judged	Correct	Wrong-match	Worth-fixing	Real-rate
TrueMend	992	87.1%	12.9%	27.6%	27.6%
Semgrep	41	87.8%	12.2%	34.1%	34.1%

Judge caveat, disclosed honestly: round 5 was Claude-judged, round 6 is glm-4.7-judged, so the 22.7%→12.9% delta reflects both the fixes and the judge change — a same-judge re-adjudication of round 5's sample was attempted but proved infeasible (its old flat-window batches exceed glm-4.7's context window). What isn't judge-dependent: raw finding counts on Dispatch dropped sharply from the fixes alone (bash-printf-echo-posix 49→0, docker-version-arg-not-used 15→0, python-mutable-class-var 34→11; 6859→6738 total, −121 false positives). TrueMend edged out Semgrep on wrong-match (12.9% vs 12.2%) — within noise, on the same corpus and judge, but near our own bar rather than under it.

Round 7 — three quarantines and the bar

One more fix, three quarantines, and a logging-recognition broadening — each a deliberate judgment call verified against Dispatch source, not a count-driven sweep. python-no-type-hints (3/8 → 0) had a negative_pattern that only excluded built-in types, so def f(common: CommonParameters) was flagged "no type hints" despite having one — broadened to recognize any name: <identifier> annotation. Three detectors were quarantined, not window-tweaked, because each one's claim is a whole-construct property no bounded line-scan can scope: css-cursor-pointer-div (4/5 wrong — whether a CSS class is interactive depends on a .vue template's @click binding, invisible to a stylesheet scan), pg_plpgsql_no_inline (4/5 wrong — "simple function" needs whole-function-body control-flow analysis a LANGUAGE plpgsql anchor can't see), and slop-formulaic-error-handling (5/7 wrong — the claim is a cross-function repetition count no single-function adjudication window can show a judge). except-no-logging (2/3 wrong → 0) was broadened to recognize traceback.print_exc() and click.secho(...) as surfacing a failure, not hiding it.

Tool	Judged	Correct	Wrong-match	Worth-fixing	Real-rate
TrueMend	976	89.1%	10.9%	27.7%	27.7%
Semgrep	41	82.9%	17.1%	39.0%	39.0%

TrueMend now clearly beats Semgrep on wrong-match (10.9% vs 17.1%), same corpus, same judge. The same-judge (glm-4.7) trajectory: 21.9% (round 5, partial) → 12.9% (round 6) → 10.9% (round 7). Raw TrueMend findings on Dispatch: 6859 → 6738 → 6468 (−391 false positives removed across rounds 6–7). No wrong-match category exceeds 3-of-8 sampled anymore — the residual is a long tail of one-off judgment calls, not a dominant cluster.

At our own wrong-match bar (10.9% vs our <10% target), and ahead of Semgrep. We're calling that honestly, not rounding it down: 10.9% is not under 10%, it's at it. One finding here was a judge error, not a detector bug, and we left it unchanged rather than "fix" it to match a wrong verdict: sqla_cascade_delete_orphan was judged wrong, but SQLAlchemy's own documentation confirms single_parent=True is genuinely required for uselist=False + delete-orphan — the finding is correct.

Still open

Real-rate is the harder, still-open bar: 27.7% vs. our >70% target, and Semgrep isn't clean on it either (39.0%, same judge) — glm-4.7 marks most correct findings on this corpus as technically-true-but-not-worth-fixing for both tools, the same judge-calibration pattern documented on our React campaign. The FastAPI-route-specific surface (5.1%, rounds 1–2 above) remains a genuine, separate win, unrelated to this gap. Closing real-rate needs judge-calibration work across the shared surface, not another FastAPI-scoped precision round — the line-scan-tractable wrong-match gap here is largely closed.