We Caught 5 of 9 Bugs. We Are Publishing the 4 We Missed.

We shipped five new detection rules this week for the bugs coding agents actually write. Before shipping them I wanted to know two numbers, not one, and the second is the number almost nobody in this market publishes.

The first number is recall: how many of the bugs do we catch? Ours is 5 out of 9.

The second is precision: how often do we yell about code that is fine? Ours is 0 false positives across 8 repositories and 177,703 lines.

I am publishing both, plus the four bugs we miss and why. The full benchmark page has the method, the per-target table, and the corpus source so you can run it yourself.

Why precision is the number I care about

Recall demos well. Point a tool at a repo, watch findings scroll, look busy. Precision is invisible until week four, when someone has dismissed the same wrong finding six times and turns the check off. After that your recall is zero, because the tool is not running.

That is not a theory about developer feelings. In a survey of 1,150 security leaders, teams reported spending 14.1 hours a week chasing false-positive alerts, and 73% said the time spent tracing alerts hurts their ability to focus on real threats.

Meanwhile the published accuracy numbers for AI code review are rough. Entelligence's 2026 benchmark ran 67 real production bugs and the best result in the whole field was 47.2% F1. Nothing cleared 50%. DeepSource's comparison puts CodeRabbit at 36.19% F1 on the OpenSSF CVE Benchmark, and Entelligence puts the same product at 33.0%. Both of those benchmarks were published by companies selling competing tools, and both rank themselves first, so treat the shape and not the decimal: independent measurements keep landing under 50% and no two of them agree.

The market's answer to that was not better accuracy. It was a new metric. "Resolution rate" counts how often a developer accepted a suggestion, which is a satisfaction score wearing a lab coat.

What we built

Two files, 52 lines, nine planted bugs. Every one is a mistake I have watched an agent make in a real repo: a query inside a loop, a write that never got awaited, a request body spread straight into an update, an empty catch, a loose equality guard, a handler with no tenant filter, a route with no auth check, a read-modify-write with no transaction, and a pagination off-by-one.

A hit only counted if it landed in the right file, on the right line, with a message describing the actual defect. "Somewhere in this file" is not a detection. If a gate cannot point at the line, nobody can act on it.

Five findings, five true positives, zero noise. The scan was complete: 2 of 2 files, nothing truncated, no degraded languages.

The four misses are the interesting part

Every one of these is reachable with a regex that would also fire on correct code, which is exactly the trade I am refusing.

IDOR is the absence of a predicate, and absence is only a defect relative to a schema. A rule for "findUnique without an orgId" fires on every legitimate lookup by primary key on earth. Missing authorization needs to know whether a guard dominates the path to the write, and that guard can live in a wrapper, a layout, or middleware. Grepping the file for the word "session" is a coin flip, and a coin flip is wrong half the time.

The off-by-one is my favorite, because it shows the limit clearly. skip: pageNum * size is correct for 0-based pages and wrong for 1-based pages. The line is not the bug. The mismatch between the line and the caller convention is the bug, and the convention is not in the file.

So the honest answer to "does CodeTruss catch IDOR" is no, not yet. The dataflow substrate that would change that is the next real build, and when it lands I will update the page rather than quietly moving the number.

Then we tried to break it

Detection on a corpus you wrote yourself proves nothing about precision, so we pointed the new rules at eight targets that were never meant to trip them: seven public SaaS starters and boilerplates plus CodeTruss itself. jonradoff/lastsaas, nextjs/saas-starter, boxyhq/saas-starter-kit, get-convex/convex-saas, ixartz/SaaS-Boilerplate, sudharsangs/nextjs-multitenant-saas-boilerplate, and fastapi/full-stack-fastapi-template.

Four fires across 177,703 lines. Zero of them false at the line.

Two were real. In get-convex/convex-saas there are two Convex mutation handlers whose whole purpose is to persist one change, and in both the database call sits as a bare statement with its promise dropped. Sibling mutations in the same file await the identical call, so the intended shape is not ambiguous. If that write rejects, the handler already returned success and the error is gone. That is the kind of bug that survives review because it looks finished. No missing brace, no smell, no failing test, just one absent keyword.

A zero on the other six only counts if you work for it. Every target that came back empty also got a canary fixture pushed through the same harness to prove the rules were actually alive, plus a manual pass over every near miss in the file. A clean sheet from an engine that quietly gave up is the worst artifact in this business, and our own history includes exactly that failure mode.

The two findings that were true and still wrong

In boxyhq/saas-starter-kit the mass-assignment rule fired at HIGH on a helper that takes data: any and passes it wholesale into an update. Every factual claim in that finding was true. The severity was not. The only caller is an allowlisted literal built from a verified Stripe webhook, so nothing in the evidence showed a request body reaching the write.

A HIGH finding asserts something the analysis had not established. Calling that "technically correct" and shipping it anyway is how tools earn the reputation I am trying to avoid. So we split the rule before release. The HIGH rule now requires actual request-flow evidence at the site, and the open-record contract case reports separately at MEDIUM, which is what the evidence supports: the type lets any caller key become a column update the day one new call site is fed by a request.

The second hit was an N+1 in a manual admin script. True shape, irrelevant cost, so script and seed paths are excluded now instead of argued with in the report.

What this does not show

It is not a comparative claim. We did not run CodeRabbit, Semgrep, or anyone else against this corpus, and the third-party numbers above are other people's published figures, cited so you can check them.

The corpus is ours, which means it measures the rules we built against the bugs we picked. Someone else's corpus would score lower and I expect that.

Zero false positives is a result on eight repositories of mostly TypeScript SaaS code, not a law of nature. Something will eventually fire wrong. When it does, the fix is the rule, and the correction gets published the same way this did.

The full benchmark has the corpus source in full, the rule IDs and CWEs, the per-target table with line counts, and the sweep protocol. Copy the two files, run them, and check my work.