BlogVerification
The future of AI code review is verification, and small teams won't get it for free
Reading a diff only goes so far. The next step is running a change against the systems it touches, and that takes infrastructure most small teams don't have.
Omer Bar-Ness · Founder
Last week Addy Osmani published how the Claude Code team thinks about verification. It's the clearest write-up I've read on how to trust code an agent wrote. We're not reinventing the wheel here. I agree with nearly all of it. This post is about the part that's easy to miss if your team is three people and a production database.
What the Claude Code team said
The post opens with a number: on the Claude Code team, about 90% of the code is written and reviewed by Claude. At that volume nobody reads every diff. So the team leans on checks that produce evidence instead. The core idea fits in one line:
“Verification loops are the building block that lets you delegate more work to Claude.”
Addy Osmani, Claude Code team · How the Claude Code team thinks about verification ↗
Without those loops, the post says, you can run one or two agents at a time, because you have to check everything yourself. With them you can run dozens.
It names three kinds of tests worth the investment. End-to-end tests come first, because they're hard to fool: the feature works or it doesn't. Property tests generate thousands of inputs to break a rule you think can't break. Differential tests run the old and new system on the same inputs and flag any difference.
Then it goes a step further: “Running the app end to end matters more than tests.” Claude drives the Chrome extension for web changes, an iOS or Android tool for mobile, or runs the service locally for backend work. A check that still fails after three rounds goes to a human. A change can't edit the check that judges it. The cheap checks (tests, build, linter) run before a pull request opens.
It's a good playbook. Read the whole thing.
Reading is not verifying
Most AI code review today, ours included, starts by reading. The reviewer gets the diff and the code around it, and reasons about it. That catches a lot: a missing null check, a leaked secret, a risky new dependency, a query built from user input.
But a reader is guessing about everything the diff talks to. Will this migration lock the orders table? Does the consumer cope with a message it has already seen? Does the new role check hold for a user who belongs to two tenants? You can't answer those by reading. You have to run it.
Unit tests don't fully close the gap either. A 2025 study of AI-generated patches on SWE-bench Verified looked at the “plausible” ones, the patches that passed the tests. 29.6% of them behaved differently from the developers' own fix. Not all of those were wrong, but some certainly were. Passing tests is evidence. It isn't proof.
Three levels of checking
Think of checks as a ladder. Each step up catches bugs the step below can't see, and each one costs more to build and keep running.
| Check | Large team | Small team |
|---|---|---|
| Reads the diff | Yes | Yes |
| Runs unit tests in CI | Yes | Usually |
| Runs against a real database and queues | Staging stack | Rarely |
| Calls real auth and third-party APIs | Sandbox accounts | Almost never |
Table 1. What a team can usually check before it merges. Our rough read, not survey data.
Real verification needs real infrastructure
Reading a diff got cheap. Writing unit tests got cheap too: an agent writes them as fast as you can ask. The Claude Code post makes the same point, that tests and specs that used to be expensive are now cheap to ask an agent for.
Real verification didn't get cheap, because the expensive part was never the typing. It's what the change has to run against:
- A database with realistic data and real migrations. Ten rows in a fixture won't show you the migration that locks a table with years of history in it.
- Kafka and queues, with ordering, retries and the same message delivered twice.
- Authentication: OAuth flows, sessions, tenants and roles.
- Third-party APIs and webhooks: payments, email, the CRM, each failing in its own way.
- Secrets the code needs to talk to all of the above.
And all of it isolated, so code nobody has reviewed yet can't touch production.
Figure 1
A large team can afford the third box. It has a staging environment, preview environments per pull request, seeded data, sandbox accounts with every vendor, and a platform team whose job is to keep all of it running. When an agent there “runs the service locally,” it boots into a world somebody already built.
Why small teams won't get it for free
A solo builder or a team of three has production and a laptop. There's no staging, or there's a staging that broke in the spring and nobody fixed. The agent can run the unit tests. It can't run the change against anything real without touching the real thing.
So the gap isn't tokens. A bigger model doesn't give you a Kafka cluster. The gap is infrastructure, and it gets wider as AI writes more of the code. More code per day means more changes that need running, and the teams writing the most AI code are often the ones with the least to run it on.
Mocks are not verification. A mock accepts whatever you told it to accept. The real broker can reject the same message, for example one over its size limit, and the test still passes.
What Docker Compose and Testcontainers do cover
To be fair, part of this is solved. Docker Compose defines and runs a multi-container app from one file. Testcontainers gives your tests “throwaway, lightweight instances of databases, message brokers, web browsers,” and has a Kafka module. A real Postgres and a real Kafka in CI is a solved problem. If you aren't doing it yet, start there.
# What one checkout service talks to, on a laptop
services:
postgres:
image: postgres:16
environment:
POSTGRES_PASSWORD: local-only
kafka:
image: apache/kafka:3.8.0
payments:
image: stripe/stripe-mock:latest
# auth: your identity provider doesn't fit in a container
That file gets you further than most small teams go. But look at where it stops:
- Auth providers. You can't run your real identity provider in Compose. A stub that always says yes tells you nothing about roles.
- Managed cloud services. LocalStack emulates a set of AWS services. It's still an emulator, not your account.
- Production-shaped data. An empty database passes every migration.
- Third-party integrations. The payments container above is stripe-mock, and its own README says it “is stateless.” Data you send it is validated, then ignored. The customer you just created isn't there on the next call.
- Upkeep. Someone has to maintain it. Image versions drift, the file grows, the seed script rots. On a team of three, that's a real slice of somebody's week.
What you can set up today
None of this means waiting. Here's what I'd set up on a small team, roughly in this order, and where each piece stops working.
-
Cheap checks first
Lint, type checks and unit tests on every pull request, plus a check on every new package before anything installs it. AI assistants invent package names. One study of code-generating models found at least 5.2% hallucinated packages on average for commercial models and 21.7% for open-source ones, with 205,474 unique made-up names. Anyone can register one of those names.
Where it stopsNone of it runs your code against anything.
-
A real database and queue in CI
Use Testcontainers or Compose. Run your migrations on every pull request, not just in production. Seed with a script that makes data look like yours: large tables, nulls where you don't expect them, non-ASCII names, rows from before the last schema change.
Where it stopsAt managed services, and at data volume you can't copy to CI.
-
A test tenant for auth
Most identity providers let you keep a separate tenant per environment. Auth0's docs say to “create a new tenant for each environment to guarantee isolation between them.” Make test users for every role, including the awkward ones: the user in two orgs, the admin who was downgraded yesterday.
Where it stopsYou're testing the test tenant's config, and it drifts from production's.
-
Recorded third-party responses
Call the vendor's sandbox once and record what comes back. WireMock can proxy to the real API and save the responses as stubs, then replay them in CI. Between your own services, contract tests check that each side still sends and accepts what the other expects.
Where it stopsRecordings show what the API said last month. Webhooks, retries and timeouts need cases you write by hand.
-
Secrets out of the agent's reach
Give the agent's environment test credentials only, scoped to the test tenant and the sandboxes. Never a production key, not even read-only. GitHub already does a version of this: “With the exception of
GITHUB_TOKEN, secrets are not passed to the runner when a workflow is triggered from a forked repository” (GitHub Docs). OWASP files the general risk under Excessive Agency: too much functionality, too many permissions, too much autonomy.Where it stopsIt's discipline, not a wall. One
.envcopied to the wrong place undoes it. -
Teach the agent to run it
Borrow the Claude Code team's idea: a short
verifyskill that boots your stack, runs the change and smoke tests the paths it touched, before the agent opens a pull request.Where it stopsIt's only as real as the stack from steps 2 to 5.
This should be a product
Put those six together and a small team has a decent setup. It also has a second job: keeping it alive. Every small team building its own copy of the same staging stack is a lot of duplicated work, and most won't do it at all.
CI went the same way. Teams used to run their own build servers. Now most small teams rent CI and never think about it. Verification against real infrastructure should go the same way: isolated, production-shaped environments a change can run in before it merges, built once and offered as a product. I don't think anyone has nailed this for small teams yet, us included.
Where MergeSafe fits today is narrower, and I'd rather say so plainly. It checks every npm and PyPI package a pull request adds for known CVEs, malicious versions, and lookalike or made-up names (that last part is our Malicious Package Guard). It verifies the bugs it finds by running a short script in an isolated sandbox. It checks CI workflows for risks like unpinned actions. And for Python projects, for now, it turns Sentry errors into fixes that pass your tests in a sandbox before they're opened. That's the cheap, early layer, not the full-stack verification this post is about.
MergeSafe checks your packages and CI workflows on every pull request, and verifies the bugs it finds in a sandbox before you merge.
Sources and further reading
- How the Claude Code team thinks about verificationclaude.dev · Addy Osmani, Nov 16, 2026
- Are “Solved Issues” in SWE-bench Really Solved Correctly? An Empirical Studyarxiv.org · Wang, Pradel, Liu, 2025
- We Have a Package for You! A Comprehensive Analysis of Package Hallucinations by Code Generating LLMsarxiv.org · Spracklen et al., 2024
- Docker Composedocs.docker.com
- Testcontainers and its Kafka moduletestcontainers.com
- LocalStack for AWS docsdocs.localstack.cloud
- stripe-mockgithub.com/stripe
- Auth0: Set up multiple environmentsauth0.com/docs
- WireMock: Record and playbackwiremock.org
- Pact docs: Introductiondocs.pact.io
- Using secrets in GitHub Actionsdocs.github.com
- LLM06:2025 Excessive Agencygenai.owasp.org