Make every PR
prove itself

The figure comes from GitHub labels applied by internal automation to agent-opened PRs; it is not the number of PRs merged, nor the company's total PR volume.

Those agents run through Nora, Abnormal's internal harness. A harness is the orchestration layer around a model: it assigns work, supplies tools, and manages context. At this volume, Nora can generate code faster than engineers can meaningfully inspect it. A row of green CI checks does not resolve the problem. It may show that unit tests passed, but nothing in the PR shows the change running against a real service.

The common response is to accelerate review with more AI reviewers, automated triage, and automatic approval for changes classified as low risk. Abnormal uses some of those techniques. In the account of Michel Chatmajian, who works on Cloud Infrastructure at Abnormal and wrote this up as a guest post on the company's Abnormal Builders blog on 2026-07-31, none of it addresses the main reason a change remains slow to review.

A reviewer looking only at the diff still has to reason through the program's behavior. So, often, the reviewer pulls the branch locally to run and inspect the remaining validation, which is slow, inefficient, and worst of all a huge context-switch for the engineer.

Abnormal changed what Nora submits. The unit of review used to be the diff, and the reviewer's job was to work out from the code whether the behavior described in the PR was the behavior the code would produce. Now the PR arrives with that behavior already exercised, and the reviewer's job is to check the artifact against the claim.

What that means concretely: every agent-opened PR must include evidence from the change running, a screenshot from a running instance, the real API response with the expected fields, the log line showing the record processed, or the new metric incrementing on a live request path.

Generating that evidence is the part that isn't easy. Every coding agent gets its own real, near-production environment with real data, hundreds of times a day. Abnormal built that environment system and calls each instance a testbox.

Proof takes two steps: define it, then produce it. A company-shared plan skill, meaning a reusable capability package available to its agents, writes a test plan into every spec, the technical design document prepared before the code. Human reviewers align on that plan before a background agent starts implementation, so the standard of proof is on the PR before the code is. The agent then configures testboxes to run pre-existing and just-in-time integration tests that produce exactly that evidence.

The template separates verification into three phases.

PR #205557 shows the UI form. An agent added a version-drift indicator to Abnormal's deployment dashboard, with a new column showing how far each cell lagged behind the fleet's target release. A browser agent started the frontend inside a testbox, connected it to state held in an isolated database, and captured the dashboard before and after the change.

In the author's experience, a screenshot can be technically valid and still prove nothing.

Interesting failures don't look like binary exceptions. Early on, a UI proof came back as a screenshot of a page that returned HTTP 200 with the right title and then never mounted. The evidence artifact existed, the comment was green, and the actual behavior was broken.

With their latest testbox harness, the screenshot tool now reports what fraction of pixels differ from the most common color, plus a preview of the text that actually rendered.

Abnormal did not find an existing system that would do that against its own service topology, so it built one.

A testbox is a real service orchestrator running in near-production configuration on a live endpoint. It gives the service under test a private copy of every datastore declared for that run. The important scaling choice is what the system does not create: it does not clone the entire platform for every change.

All agents share one running base environment, and each testbox overrides only the service being changed.

Trace one request through it. A change to Signal Enrichment leaves Message Intake, Detection Engine and Verdict shared. The request enters the shared pipeline, runs through the shared intake, and is bent at exactly one hop into the agent's own instance, which talks to its own private datastores. Then it rejoins the shared path and comes out the other end at Verdict.

The variations follow from that shape. Another agent can override Detection Engine instead, bending a different hop. A change spanning services can replace both Signal Enrichment and Detection Engine and attach its own seeded database, which the source calls a multi-service slice.

That division determines the economics of the system. The agent receives the real surrounding topology without paying to reproduce every service in it. You pay for that hop and its datastores, not for a copy of the platform, which the author calls the difference between a few hundred of these a day being routine and being unthinkable.

The redirect costs the service nothing, because Abnormal's services already resolve dependencies through an internal alias scheme rather than hardcoded connection strings. A testbox rewrites the destination behind an alias. The service itself runs unmodified code while its dependency path points at the testbox resource.

That indirection is also where it bites you. A redirect only works if every consumer of the config layer honors it, and the ones that don't fail silently.

The services entry points a dotted service identifier at the PR branch via git_ref; the source calls that identifier a pac and does not define the scheme further. A components entry declares the database, while the engine and version are resolved from the manifest. Two startup_flows prepare the state: init_schema creates the sandbox database, strategy: reflect mirrors the deployed schema, and run_sql then inserts rows into custom_rule_match_log, shaping the exact rows the proof needs.

An agent can compose new infrastructure and reuse existing infrastructure at runtime. If the declaration is wrong or the environment becomes corrupted, the agent can discard that instance and respawn it with another tool call.

Integration proof also requires real data. Postgres, OpenSearch, Kafka, DynamoDB, and Redis run as their real engines in containers, per testbox, in seconds, at the same versions used by the deployed service. What gets seeded is a per-store decision, and it isn't always a copy.

In the plan on the PR, evaluated_at is an existing column that was never selected or returned through the API, while evaluation_latency_ms is computed server-side in the handler from evaluated_at - message_sent_at rather than stored.

The returned values had to match the stored data exactly. The latency calculation had to remain correct for every case, including a zero delta. Pagination also had to continue working, including the has_more behavior.

The agent launched the real service from its PR branch against a real Postgres instance. It seeded four rows with known timestamps, then called the endpoint over gRPC with a small page size, limit=2. Each returned field matched its seeded row. The latency calculations were correct, and pagination returned every row exactly once.

The evidence also carries a note on the boot path. Out of the box the sandbox pod dialed the shared test database instead of the cluster's own Postgres, so the seeded rule wasn't found. The agent re-launched the same binary with a local-DB override so it pointed at the sandbox Postgres: same binary, same code path, just aimed at the seeded database. This exercises the change against a real Postgres. Here the wrong database was caught because the seeded rule was missing.

One assertion row appeared to be missing evaluation_latency_ms. The evidence annotated it as omitted (proto3 default 0), compared it with the expected value 0, and marked it as passing. Under proto3, a scalar field set to its default value, including an int64 value of 0, is omitted from the encoded response. The absence of the field on the wire was therefore the correct representation of zero latency, not a failed calculation.

The evidence carries the seeded value, the expected result, the encoded response, and the assertion connecting them, including the reason an irregular-looking row satisfies the protocol.

The completed run displayed Testbox · done — 15m49s · 76 steps. Beneath it were the execution records: the grpcurl request, the operation that seeded four rows with known deltas, and the deletion of the sandbox cluster.

There is no required job that runs testbox checks at a fixed point in the pipeline. It is a tool surface, and any harness holding an agent loop can pick it up.

An engineer can comment /testbox on a PR and receive a run against the diff currently under review. A background agent working on a ticket can invoke the same tools in the middle of its loop, inspect a failure, patch the code, and run the check again before a human sees the change. The interface is also available through Slack, Jira, and local coding agent CLIs, placing the trigger inside the systems where work is already happening.

Speed keeps that loop usable. Deploying a single service through the full CI/CD system can take 20 to 30 minutes. A testbox can start the same service against a real database and its dependencies in less than 5 minutes, often sooner than the system can be made operational on a local laptop. Because the harness persists, each successive check reuses it rather than paying the setup cost again, so the loop gets cheaper the longer the agent works.

When an incident or deployment failure exposes a missing case, Abnormal writes the miss back into both the testbox checks and the planning skill. A later change with the same shape begins with that condition already present in its proof requirements. The failure moves from something discovered after merge to something the agent must exercise before requesting review.

Over time, in the author's words, the gaps that only production used to catch move earlier, with the test environment's production parity iteratively and automatically improving.

By Abnormal's account, adoption has been steeper than planned: agents used over 3,000 testboxes in the last 30 days across 64 engineering teams. The testbox and team figures come from testbox creation records for July 2026, while the separate agent PR volume is counted from GitHub labels applied by internal automation to agent-opened PRs. The two figures are counted from different records and neither is derived from the other.