Detection benchmark · Adversarial corpus · Published 2026-08-06

Five of nine caught. Zero false positives.

A first-pass verification gate is only worth installing if it never cries wolf. So before we shipped five new rules for the bugs AI agents actually write, we built a corpus of nine of them, measured exactly how many the engine sees, and then pointed those rules at eight real repositories to find out how often they fire when nothing is wrong. This page is the whole method, the whole result, and the four bugs we do not catch.

Detected: 5 / 9
False positives: 0
Repositories swept: 8
Lines swept: 177,703

§ 01 — Thesis

A gate that cries wolf gets uninstalled. That makes precision the product.

Every automated reviewer trades recall against precision, and almost every vendor optimizes the side that demos well. Recall is easy to show on stage: point the tool at a repository, watch findings scroll. Precision is invisible until the fourth week, when a developer has dismissed the same wrong finding six times and turns the check off. At that point the tool’s recall is zero, because it is not running.

This is not our theory about developer psychology, it is the security industry’s measured cost. In a survey of 1,150 security leaders, teams reported spending 14.1 hours a week chasing false-positive alerts, and 73% said the time spent tracing alerts hurts their ability to focus on real threats.

Meanwhile the published accuracy numbers for AI code review are not good. Entelligence’s 2026 benchmark of 67 real production bugs put the field’s best result at 47.2% F1— nothing in it cleared 50%. DeepSource’s comparison on the OpenSSF CVE Benchmark reports 36.19% F1 for CodeRabbit; Entelligence reports 33.0% for the same product. Both benchmarks were published by companies selling competing tools, and each ranks itself first, which is the honest reason to quote the shape rather than the decimal: independent measurements keep landing under 50% F1 and no two agree. The market’s response was not higher accuracy, it was a new metric — “resolution rate,” which counts how often a developer accepted a suggestion rather than how often the tool was right.

So we set the bar where a gate has to be set. A finding that fires must be true at the line it names, or the rule does not ship. Recall is something we grow honestly, one rule at a time, in public. This page reports both halves.

Sources, in order: Illumio / Vitreous World alert survey (1,150 security leaders) · Entelligence 2026 code review benchmark · DeepSource AI code review tools comparison · DeepSource — notes on AI code review benchmarks

§ 02 — Method

How the corpus was built and what counted as a hit.

The corpus

Two files, 52 lines, nine planted defects. Every one is a mistake we have watched coding agents make in real repositories: the agent writes a loop that reads the related record per item, forgets an await on a write, spreads a request body into an update, wraps a call in an empty catch, compares with ==, or ships a handler with no tenant filter and no authorization check.

The corpus is deliberately small and deliberately ours. It is not a CVE set and it is not sampled from the wild, so it measures what our rules see on the bug classes we chose to target — nothing more. The counterweight to that self-selection is the sweep in §05, which is not ours at all.

What “detected” required

A hit had to land in the right file, on the right line, with a message that described the actual defect. “Somewhere in this file” is not a detection, because a gate that cannot point at the line cannot be acted on. All five hits below met that bar; each rule fired exactly once.

The scan also had to be complete before its output counted: two of two files scanned, no truncation, no degraded languages, no budget or resource limit reached. A green sheet from a scan that quietly stopped early is a lie, and our own history includes exactly that failure mode.

The refute-first protocol

Each of the eight sweep targets was worked by a separate reviewer whose job was to break the result, not confirm it. Four gates had to pass before any target could be recorded.

01
Prove the scan was complete
Every run records files scanned versus files offered, truncation, degraded languages, and budget state. A partial scan cannot produce a clean sheet, because a clean sheet from an engine that quietly gave up is the worst artifact in this business.
02
Prove the rules were alive
Zero findings and a broken rule registration look identical from the outside. So every target that returned zero also got a canary fixture pushed through the identical harness path; all five rules had to fire at their planted lines before the zero was accepted.
03
Try to refute the zero by hand
For each rule we searched the target for its candidate shape and read every near miss: catch clauses, loose comparisons, query sites in loops, write payloads. A zero is only publishable as a set of adjudicated true negatives, never as an absence of evidence.
04
Adjudicate every fire against the source
Each finding was opened at its line and argued both ways. The verdicts were true positive, defensible, or false positive — and defensible was treated as debt to be fixed before shipping, not as a win.

§ 03 — Corpus result

Five of nine, each at the exact line. Four misses, named.

Bug class	Site	Rule	CWE	Severity	Result
N+1 queryA findUnique for the related customer inside a for-loop over invoices.	invoices.ts:13	db-call-in-loop	CWE-1050	MEDIUM	CAUGHT
Floating database writeAn audit-log create whose promise is discarded — never awaited, returned, or handled.	invoices.ts:21	unawaited-persistence	CWE-252	MEDIUM	CAUGHT
Mass assignmentAn open-record payload spread wholesale into an update. Reported as the contract defect the evidence supports, not as a request-flow claim.	invoices.ts:26	open-record-write	CWE-915	MEDIUM	CAUGHT
Swallowed errorAn empty catch around an outbound fetch, with no logging and no comment.	invoices.ts:39	swallowed-error	CWE-1069	LOW	CAUGHT
Loose equalityamount == 0, where the string '0' and the empty string both pass the guard.	invoices.ts:49	loose-equality	CWE-697	MEDIUM	CAUGHT
IDORAn invoice fetched by id with no tenant filter, so any user can read any org.	invoices.ts:4	—	CWE-639	—	MISSED
Read-modify-write raceA balance read, incremented in memory, then written back with no transaction.	invoices.ts:30	—	CWE-362	—	MISSED
Pagination off-by-oneskip: pageNum * size, which drops one record when pages are 1-based.	invoices.ts:43	—	CWE-193	—	MISSED
Missing authorizationA mutating POST route handler with no session or role check anywhere on the path.	route.ts:4	—	CWE-862	—	MISSED

Five findings, five true positives, zero false positives on the corpus. The scan was complete: two of two files, no truncation, no degraded languages. Two further findings came from repository-structure analyzers (missing README, no CI pipeline) and are unrelated to the planted bugs — they are neither counted as hits nor as noise.

One detail worth stating plainly, because it is the difference between a benchmark and a brochure: the mass-assignment bug is reported by open-record-write at MEDIUM, not by the HIGH request-flow rule. In the corpus the payload arrives as a Record<string, unknown> parameter, so the evidence at that line supports a contract claim, not a proven request flow. §07 explains why that distinction exists.

§ 04 — Misses

The four we do not catch, and the analysis each one actually needs.

These are the most interesting rows in the table. Each of the four is reachable with a regex that would also fire on correct code — which is exactly the trade this benchmark exists to refuse. Here is what each one really needs.

IDOR — no tenant filter on a read

The defect is the absence of a predicate, and absence is only a defect relative to a schema. A syntactic rule for "findUnique without an orgId" fires on every legitimate lookup by primary key in every repository on earth. Deciding this correctly means knowing which columns carry tenancy for this schema, whether the caller already constrained the query, and whether the handler is tenant-scoped upstream. That is source-to-sink dataflow plus schema ownership, not a node shape.

Missing authorization on a mutating route

The honest question is whether an authorization guard dominates every path that reaches the write — a guard can live in the handler, a wrapper, a layout, proxy middleware, or the framework config. Grepping the file for the word "session" is a coin flip, and a coin flip is a false positive half the time. This needs route enumeration and guard-reachability analysis over the call graph.

Read-modify-write race

Two statements are only a race if they touch the same row and no transaction or atomic-update construct encloses them. That is an inter-statement dataflow property over a receiver identity the engine does not yet track. It is the most rule-able of the four, and it is the next one we intend to close.

Pagination off-by-one

skip: pageNum * size is correct for 0-based pages and wrong for 1-based pages. The line is not the bug; the mismatch between the line and the caller convention is the bug, and the convention is not in the file. Closing this needs the caller contract — invariants or property tests, not a pattern.

Two of the four — IDOR and missing authorization — are the reason a dataflow substrate sits at the top of the engine roadmap: route and handler enumeration, guard reachability, and parameter-to-query taint. When those land, these rows change, and this page changes with them. Until then the honest answer to “does CodeTruss catch IDOR?” is no.

§ 05 — Adversarial sweep

Eight repositories, 177,703 lines, zero false positives.

Detection on a corpus you wrote yourself proves almost nothing about precision. The real test is pointing new rules at code that was never meant to trip them. We swept seven public SaaS starters and boilerplates — chosen because they are dense with exactly the shapes these rules look at — plus CodeTruss itself, and read every single fire by hand against its source.

Target	Files	LOC	Scanned	New-rule fires	Adjudication
jonradoff/lastsaas130 catch clauses, every one handling or documenting. Zero loose comparisons in TypeScript. Go backend sits outside SAST language coverage and the report says so.	260	48,692	187 / 187	0	No fires
nextjs/saas-starterEight catch clauses, all logging and responding. Every Drizzle write awaited, returned, or sitting in a Promise.all argument position the anchor structurally excludes.	55	3,576	42 / 42	0	No fires
boxyhq/saas-starter-kitThe two hits that produced the severity split. Eight-plus loose comparisons elsewhere in the repo were correctly suppressed against non-coercion-prone literals (POST, per_unit, tiered).	327	17,157	261 / 261	2	Both true, both re-priced
get-convex/convex-saasTwo discarded ctx.db.patch promises in convex/app.ts. Per-key queries inside asyncMap callbacks were correctly not flagged as N+1 by the function-boundary gate.	81	5,131	58 / 58	2	Two true positives
ixartz/SaaS-BoilerplateWeak evidence, and we say so: the repository contains no candidate shapes at all — no try/catch, no loose equality, no query sites — so it exercised almost none of the suppression logic.	151	3,465	88 / 88	0	No fires
sudharsangs/nextjs-multitenant-saas-boilerplateThe one empty catch wraps a clipboard write and is suppressed by the cleanup carve-out. A real per-item query inside Promise.all was left alone by the function-boundary gate — a deliberate miss, logged as one.	100	8,660	79 / 79	0	No fires
fastapi/full-stack-fastapi-templateThe Python backend idioms these rules were designed around — session.commit(), Model.model_validate — were correctly left alone. No except/pass anywhere in the backend.	228	10,069	105 / 105	0	No fires
CodeTruss (this repository)All 19 SAST findings came from older rules. Four near-miss sites held: two best-effort cleanup catches in the CLI, and two bounded-lookback loops with early exits.	695	80,953	380 / 380	0	No fires

Four fires across 177,703 lines. Two were clean true positives in a public template (§06). Two were factually true but priced wrong, and rather than publish around them we changed the engine (§07). No fire in the sweep was false at its line.

Six targets returned zero. We do not present those as wins by themselves — a zero earns nothing until you have proved the rules were alive and read every near miss in the file. Two targets contain almost no candidate shapes at all, and the table says so instead of counting them as evidence.

§ 06 — Case study

A real dropped write, in a template thousands of people fork.

The sweep’s two true positives both landed in get-convex/convex-saas, at convex/app.ts lines 108 and 119. Both are Convex mutation handlers whose entire purpose is to persist one change, and in both the database call is written as a bare statement with its promise discarded.

What makes these adjudicable rather than arguable is the evidence sitting in the same file: sibling mutations in that module await the identical call. The intended shape is unambiguous, and the two sites do not match it. If the write rejects — schema validation, a conflict, a transient failure — the handler has already returned success and the error is discarded.

This is the class of defect that survives review precisely because it looks finished. Nothing about the line is ugly. There is no missing brace, no obvious smell, no failing test. One keyword is absent, and a write silently stops being guaranteed.

Reported here as an analysis result on public MIT-licensed source at the commit we scanned, not as a vulnerability disclosure or a criticism of the project. Convex ships a good template; this is what a first-pass gate is for.

§ 07 — Severity honesty

Two findings were true and still wrong. So we split the rule.

In boxyhq/saas-starter-kit the mass-assignment rule fired at HIGH on models/subscription.ts:40, where a helper takes data: any and passes it wholesale into an update. Every factual claim in that finding was true. The severity was not: the only caller is an allowlisted literal built from a verified Stripe webhook, so nothing in the evidence showed a request body reaching the write.

A HIGH finding asserts something the analysis had not established. Under our own standard that is a defect in the rule, even though the pattern claim held. Calling it “technically correct” and shipping is how tools earn the reputation this benchmark exists to avoid.

So the rule was split before release. The HIGH rule now requires actual request-flow evidence at the site. The open-record contract — a payload typed any or Record<string, unknown> written straight into a record — reports separately at MEDIUM, which is what that evidence actually supports: the contract invites mass assignment the day one new call site is fed by a request.

The second hit was an N+1 in a manual admin script. True shape, irrelevant cost, so script and seed paths are now excluded from that rule rather than argued with in the report.

Before / after · models/subscription.ts:40

Before the split: HIGH · Mass assignment: request body written wholesale to a database record
After the split: MEDIUM · Database write helper accepts an open-record payload

Severity is a claim about evidence. It never gets to run ahead of the analysis that produced it.

§ 08 — Reproducibility

The corpus, in full. Copy it and run it yourself.

A benchmark you cannot re-run is marketing. Both corpus files are printed here complete — 52 lines total. Drop them into a repository at these paths, scan it, and you should see five findings at lines 13, 21, 26, 39, and 49 of src/lib/invoices.ts and nothing in src/app/api/invoices/route.ts.

src/lib/invoices.ts

import { db } from './db'

// BUG 1 (IDOR): no orgId filter — any user reads any org's invoice
export async function getInvoice(invoiceId: string) {
  return db.invoice.findUnique({ where: { id: invoiceId } })
}

// BUG 2 (N+1): query inside a loop
export async function listWithCustomers(orgId: string) {
  const invoices = await db.invoice.findMany({ where: { orgId } })
  const out = []
  for (const inv of invoices) {
    const customer = await db.customer.findUnique({ where: { id: inv.customerId } })
    out.push({ ...inv, customer })
  }
  return out
}

// BUG 3 (floating promise): not awaited — write silently lost on error
export function recordAudit(orgId: string, action: string) {
  db.auditLog.create({ data: { orgId, action } })
}

// BUG 4 (mass assignment): spreads raw client input into an update
export async function updateInvoice(id: string, body: Record<string, unknown>) {
  return db.invoice.update({ where: { id }, data: { ...body } })
}

// BUG 5 (race): read-modify-write without a transaction
export async function incrementBalance(id: string, amount: number) {
  const inv = await db.invoice.findUnique({ where: { id } })
  return db.invoice.update({ where: { id }, data: { balance: inv.balance + amount } })
}

// BUG 6 (swallowed error): empty catch hides failures
export async function sendReceipt(id: string) {
  try {
    await fetch(`https://mail.example.com/send?invoice=${id}`)
  } catch {}
}

// BUG 7 (off-by-one): page 1 skips a record
export async function page(orgId: string, pageNum: number, size = 20) {
  return db.invoice.findMany({ where: { orgId }, skip: pageNum * size, take: size })
}

// BUG 8 (loose equality): '0' == 0 is true, so zero-amount passes
export function isPaid(amount: unknown) {
  return amount == 0
}

src/app/api/invoices/route.ts

import { getInvoice, updateInvoice } from '@/lib/invoices'

// BUG 9 (missing authz): no session/role check on a mutating route
export async function POST(req: Request) {
  const body = await req.json()
  return Response.json(await updateInvoice(body.id, body))
}

export async function GET(req: Request) {
  const { searchParams } = new URL(req.url)
  return Response.json(await getInvoice(searchParams.get('id')!))
}

The five rules

db-call-in-loop: CWE-1050 · MEDIUM · awaited DB read in a loop whose arguments change per iteration
unawaited-persistence: CWE-252 · MEDIUM · a write whose promise is discarded entirely
mass-assignment: CWE-915 · HIGH · a request object reaching a write payload
open-record-write: CWE-915 · MEDIUM · an any/Record payload written wholesale to a record
swallowed-error: CWE-1069 · LOW · an empty catch with no handling, logging, or comment
loose-equality: CWE-697 · MEDIUM · == or != against a coercion-prone literal

The silences are the design

Each rule carries explicit carve-outs, and they are why the sweep came back clean: x == null is never flagged; a comment inside a catch is documented intent; best-effort cleanup is exempt; a function boundary between the loop and the query means the query belongs to a callback; bounded loops with early exits are searches, not N+1; and script, seed, and migration paths are out of scope for cost rules.

§ 09 — Limits

What this benchmark does not show.

It is not a comparative claim.We did not run CodeRabbit, Semgrep, Snyk, Bugbot, or any other tool against this corpus. The third-party accuracy numbers quoted in §01 are those vendors’ and researchers’ published figures, cited so you can check them; they are not measurements we made, and they were not produced on this corpus. Nothing here says we score higher than anyone.

The corpus is ours. Nine bug classes we selected, planted in code we wrote. It measures the rules we built against the bugs we chose to target. A corpus written by someone else would produce a different detection rate, and the honest expectation is that it would be lower.

Five of nine is a low recall number, and it is the number. Four of the nine bug classes here are invisible to this engine today, including the two with the worst security consequences. If your risk is IDOR, this is not yet the tool that finds it.

Zero false positives is a result on eight repositories, not a law of nature. It is 177,703 lines of mostly TypeScript SaaS code. Alien idioms, other languages, and other frameworks will eventually produce a fire we have to argue with — and our own scanning history has included silent-failure modes we found and fixed. When precision breaks, the fix is the rule, and the correction gets published like this page did.

Detection is not the whole verification story. These five rules run in the hosted analysis. The local CLI runs deterministic registry analyzers and discloses which passes did not apply rather than implying coverage it does not have. Where a run could not analyze something, the receipt says so.

Everything here is a snapshot. The public repositories were scanned at the commits available on 2026-08-06; upstream code changes. Findings are analysis results on public source, not vulnerability disclosures, and no project named here is affiliated with or endorsing CodeTruss.

Run the same gate on your own repository.

The CLI is free and runs locally: it classifies every changed path, runs the deterministic analyzers and your own verification commands, and leaves a signed receipt that records what was checked and what was not. The hosted audit adds the whole-repository pass these five rules run in.

Install the CLI Audit a public repo Read why we published the misses