← all writing

The 56% Problem

Models keep getting smarter. The industry's security pass rate hasn't moved. Here's what we actually found when we routed around it instead of waiting on it.

Veracode put out their yearly GenAI code security report a few weeks back, and the number that stuck with me is 56%. That's the pass rate for AI-generated code on basic security tests, across a hundred-plus models, and it's barely moved since last year's 55%.

AI-generated code: industry security pass rate
2025 Report 1 55%
2025 Report 2 54%
Spring 2026 55%
Summer 2026 56%
Source: Veracode 2026 GenAI Code Security Report, four testing snapshots.

Every model got smarter. Every benchmark that measures "does this code work" went up. The number that measures "does this code work safely" didn't move. I think a lot of people assumed those were the same problem, that a model good enough to write clean code would just naturally write secure code too. It doesn't work that way.

Writing code that runs and writing code that doesn't leak your users' data are different skills, and only one of them scales with model size.

Here's the thing, and I want to be precise instead of hand-wavy about it: TrueMend doesn't fix this by having a smarter model watch the model. It fixes it by not needing a model at all for most of what it catches. The canonical description is a deterministic-first code quality audit and remediation engine, and that word "deterministic" is doing real work. The core analyzer is pure static analysis, AST-based, no LLM in the loop. It produces a metrics JSON. That's it.

Where a finding actually goes

Every pattern TrueMend catches gets routed into one of three tiers, and this is the real differentiator, not the pattern count:

Where a finding goes: the three-tier fix taxonomy
Deterministic 38%
Cataloged refactor 56.8%
Structural rewrite 5%
4,709 catalogued patterns, split by fix tier. Source: internal facts.json, current as of this post.

38% of what's catalogued is deterministic: a mechanical AST transform, verified, no model touches it, and it's free. Just under 57% is a cataloged refactor, a named, bounded procedure that a client-side LLM executes against constraints TrueMend hands it, never an open-ended "fix this." And 5% is structural, the kind of change that stays a human's call, always, no exceptions.

That's the actual answer to the 56% problem, in miniature. A chunk of what's wrong with AI-generated code doesn't need a model to fix, it needs a transform that's provably correct. Another chunk needs a model, but a model working inside a fence instead of freelancing. And a chunk needs a person, permanently, because some decisions shouldn't be automated at all.

What a deterministic fix looks like — caught by an AST transform, applied with no LLM call:

// before
if (user.role !== undefined) {
  if (user.role !== null) {
    processRole(user.role);
  }
}

// after patch_engine
if (user.role != null) {
  processRole(user.role);
}

Nobody needed to prompt a model to collapse that nesting. It's a pattern with one provably correct rewrite, so it's handled by a transform, verified, and moved on from. That's what "deterministic" buys you: the 38% doesn't argue with itself, doesn't hallucinate a fix that looks plausible and isn't, and doesn't cost anything per call.

The Veracode number is the industry looking at "smarter model" and "safer output" as the same lever. They're not. Model quality moves one number. A fix taxonomy that only lets a model touch what it's actually good at moves a different one.

← all writingnext: how the engine works →