TrueMend vs. actionlint on GitHub Actions

actionlint is the purpose-built, single-ruleset linter for GitHub Actions workflow YAML — a corpus of 165 real workflows, flattened from eleven benchmark repos' .github/workflows/ directories, gave us a fair fight on its home turf.

The baseline was a confirmed loss

First measurement: TrueMend 55.7% wrong-match vs. actionlint's 44.4% — independently judged, not self-reported, and a clear loss. We published that number to ourselves before touching anything, the same discipline as every other campaign in this ledger.

Root cause: two detector families that don't belong in this file type

The single biggest lever, by far: 66 Kubernetes detectors and 24 Docker Compose detectors had no file_content_requires gate, so they fired on a GitHub Actions workflow's matrix.image:, container:, and services: blocks as if those were real Kubernetes manifests or Compose files — k8s-kubelinter-no-service-account alone went 8-for-8 wrong, every hit on a matrix.image: line that isn't a Kubernetes resource at all. Adding apiVersion: and (?m)^services: content gates before those detectors fire silenced 496 false positives outright — 1,079 findings down to 575 — with no loss of recall on files that are actually Kubernetes or Compose manifests.

Seven more detectors had a second, distinct problem: a premise that simply doesn't hold for a workflow trigger context. yaml-github-actions-secrets-plaintext was flagging default values like POSTGRES_PASSWORD: postgres that are container bootstrap conventions, not leaked secrets. ci-token-write-permissions treated any write-scoped permission as broad when only contents: write specifically is the risky one. Four detectors — checks for status checks, PR comments, conditional gating, and environment protection — can't actually be verified from the trigger event alone, so we disabled them as premise errors rather than keep a claim we couldn't back. One more got its Dockerfile anchor narrowed, and a matrix-include threshold got raised to reduce noise on small, legitimate matrices.

Round 1 result — a 29-point swing

TrueMend: 11.6% wrong-match. actionlint: 40.0%. 112 TrueMend findings and 35 actionlint findings, judged blind, same-session head-to-head. Worth-fixing narrowed from the baseline's 3.7%/30.6% gap to 17.0%/25.7% — closer, though still trailing.

What we weren't claiming in round 1

actionlint's expression, action, and syntax-check rules catch real workflow-expression correctness bugs — invalid property references, deprecated action versions — that a line-scan detector can't structurally express without a full workflow-expression AST. The same shape of gap shows up in the C# campaign's XML-doc case. In round 1, TrueMend won the headline wrong-match metric decisively; actionlint still found more genuinely worth-fixing correctness bugs on this corpus, and we said so rather than only publishing the number that flattered us.

Result — round 2, a fresh full-pool sample

TrueMend: 19.5% wrong-match. actionlint: 40.0% — round 1's number, carried forward, not re-measured this round. A parallel fix pass targeted the top wrong-match categories still open after round 1, then a fresh, from-scratch full-pool blind sample re-certified the result: 77 TrueMend findings judged. Four more categories fixed this round: devops-hardcoded-secret (well-known CI credential literal exclusion), continue-on-error (outcome-check negation logic), ci-no-release-please (release-event gate added), and a second pass on ci-token-write-permissions — round 1 fixed the broad-write-scope premise, round 2 added a push/commit-bot and reusable-workflow exclusion the first pass didn't cover. A new regression test file, tests/regression/test_ghactions_fp_regression.py, locks in all four — 10/10 passing.

The wrong-match number moved from 11.6% to 19.5% — worse on the raw figure. That's the honest number, and it's not a reversal: round 1's fixes were independently re-verified this round and still hold. What moved is sample composition — round 2 is a smaller, differently-composed fresh pool (77 judged vs. 112), pulling from parts of the detector surface round 1's sample didn't touch, the same pattern seen across every re-certified target in this sweep (Bash, Java, Go, SCSS). The more meaningful signal this round is real-rate: worth-fixing more than doubled, from 17.0% to 40.3%.

Still open

actionlint has not been re-run this round — a true apples-to-apples round-2 comparison against a fresh actionlint measurement doesn't exist yet. TrueMend's round-2 wrong-match, 19.5%, remains above the 10% gold-standard bar this ledger holds every target to. Both are open, not closed.