Cognition Says Reviewers Work Better Blind. Mine Didn't.

The previous post left two real bugs sitting in two files: a receipt module split between two isolated subagents, one half that doesn’t compile, and a fixed version that compiles, runs, and is silently wrong by a cent on the right input. This post asks the obvious next question — would a reviewer have caught either one — and tests a specific, counterintuitive claim about how to build that reviewer.

Cognition’s own numbers: their review agent “catches an average of 2 bugs per PR, of which roughly 58% are severe.”¹ The part that’s actually counterintuitive is how they built it — they “found this technique to work best when the coding and review agents do not share any context beforehand,” and they attribute this to context rot, the documented tendency for a model to reason worse as its context window fills up.¹ A reviewer that never saw the coder’s reasoning stays short and sharp. A reviewer fed the coder’s own accumulated exploration inherits the bloat, and reasons worse for it.

I ran that exact comparison on the two buggy files from the last post. It came out backwards.

The setup

Same two files, unmodified: lineitems.ts (no export statement — the compile-breaking bug) and total.ts (rounds the subtotal once at the end, while lineitems.ts rounds every line individually — the one-cent bug). Two fresh general-purpose subagents, launched in the same message so neither could see the other, each told to review the pair for “bugs, integration risks, or correctness issues” and free to compile or run the code if it wanted to. Identical instructions, identical files. The only difference: one reviewer also got a 210-word, 1,492-character handoff note — the two original coder agents’ own summaries of what they built, quoted verbatim from the last post.² Full experiment details are in the methodology note.³

flowchart TB
F["lineitems.ts + total.ts<br/>(unmodified from previous post)"]
H["handoff note<br/>210 words, 1,492 chars<br/>the coders' own self-reports"]
F --> C["Cold reviewer<br/>no context<br/>60,093 tokens"]
H --> W["Warm reviewer<br/>reads handoff note first<br/>62,512 tokens"]
F --> W
C --> RC["6 findings<br/>1 blocker / 2 major / 3 minor"]
W --> RW["5 findings<br/>1 blocker / 2 major / 2 minor"]

What both of them found

Both reviewers, unprompted, went further than reading: each set up its own scratch tsc project and ran the files with self-chosen test values — nobody told either one to compile or execute anything. Both landed on the same blocker independently: lineitems.ts exports nothing, so tsc fails with TS2306 the instant anything tries to import it — the identical failure the previous post found by hand.

Both also flagged the same underlying defect in total.ts — its $ column doesn’t line up across the Subtotal/Tax/TOTAL lines, because the padding before each $ is a fixed literal instead of computed from the label’s length. They disagreed on how much it mattered: the cold reviewer called it minor, the warm reviewer called it major. Same defect, same evidence, different verdict — a reminder that an LLM-assigned severity label is a judgment call, not a measurement.

That’s the entire overlap. Two shared findings out of nine distinct ones across both reports.

Where each reviewer’s own testing led it

Neither reviewer was told to look for a rounding bug, or told anything about money at all beyond “total/tax calculator.” Both invented their own adversarial inputs — this is worth sitting with, because the last post’s whole finding was that nobody thinks to test adversarial values unless something specifically points them there. Here, two reviewer agents did it on their own initiative, unprompted, simply because “review this for correctness” implied it to both of them.

They just tested different axes. The cold reviewer varied the tax rate — 6%, 8.25%, 0% — checking total.ts in isolation. That never touches the bug: the rounding mismatch lives in how many items get summed and when each one rounds, not in the tax percentage. The warm reviewer, having just read that Developer A’s example used two items and Developer B’s used a different two, tried a cart with fractional quantities — and that’s the exact shape of input that exposes the defect. Three lines each rounding to $0.01 from a raw $0.005 print a visible $0.03; total.ts sums the same raw values first and rounds once, printing $0.01. A receipt that visibly doesn’t add up, caught with the reviewer’s own test case, not the one from the previous post.

Both reviewers verified. Only one verified in the right place.

Why this probably isn’t a real rebuttal

Cognition’s explanatory mechanism is specific: context rot — a model reasoning worse as its window fills with the coder’s own accumulated working history.¹ That’s not what I gave the warm reviewer. The handoff note is 210 words. It is not a trace, a transcript, or an accumulated exploration — the previous post in this series measured that a subagent’s actual working history isn’t even retrievable on this harness: 0-byte transcript files, 0 sidechain records, for every subagent it checked.⁴ The only thing that ever crosses a subagent boundary here is a short result string. A 210-word handoff note is that short result string — it’s close to the maximum context a reviewer could realistically receive on this harness, not a sliver of a much larger trace that got rotted down.

So the honest reading isn’t “Cognition is wrong.” It’s narrower: the two conditions aren’t the same experiment. Cognition’s warning is about inheriting a coder’s own reasoning at real length. What I tested is whether a short, comparative summary — both developers’ stated assumptions, side by side — helps or hurts. In this one case, seeing both assumption sets side by side pointed the reviewer straight at the seam between them, which is exactly where the bug was hiding. That’s a plausible, different mechanism from context rot, and it’s the one this specific bug was built to need: it only exists at the boundary between two files, and the boundary is the one thing neither file’s own code makes visible.

What it cost

The handoff note cost the warm reviewer 2,419 extra tokens over the cold one — about 4% more, for the file that mattered. Stack the whole pipeline: 98,856 tokens for the two coders in the previous post, plus 60,093 and 62,512 for the two reviewers here, comes to 221,461 tokens to write and review a two-function receipt module that a human could write correctly in about the same time it took to read this paragraph. Whether that’s worth it depends entirely on what the bug would have cost left in — and a receipt that silently disagrees with itself by a cent is exactly the kind of bug that’s cheap to write, cheap to miss, and expensive to explain to a customer.

Trade-offs

N = 1. One pair of files, one pair of reviewers, one model generation. This is a demonstration that the effect can point the other way, not a replacement for Cognition’s own benchmark — which is drawn from many more PRs than one.

Severity is a self-report. Both reviewers rated their own findings’ severity with no independent check. The one place they disagreed on severity for the identical defect (minor vs. major, same $-alignment bug) is the clearest evidence that this number is a judgment call, not a fact — worth weighting accordingly anywhere an agent’s severity rating drives a triage decision.

Overlap was worse than either reviewer alone. Running both reviewers found more total defects (9) than either one did (6, then 5) — but two reviewers cost roughly double, and 78% of what they found was reviewer-specific. If you can only afford one reviewer, this one trial doesn’t tell you which kind to pick; it tells you that picking either alone leaves real bugs on the table.

The type of context is the variable, not its presence. A short, structured, two-sided handoff note is a different object from an inherited single-writer trace. Cognition’s context-rot mechanism is about the latter; this experiment tested the former, because the former is the only kind of context this harness can actually hand a reviewer.

Takeaways

• The reviewer with context caught the bug the blind one missed. On these two files, “share no context” and “review well” pointed in opposite directions — the opposite of Cognition’s own finding, on a much smaller and different kind of context than theirs.
• A 210-word handoff note is a plausible ceiling for “coder context” on this harness, not a floor. Full agent traces aren’t retrievable here — post 2 measured 0-byte transcripts. The most context any reviewer here could get is a short summary, and that’s exactly what was tested.
• Both reviewers verified by execution, unprompted — and still only found what they thought to test. The cold reviewer varied tax rate; the warm reviewer, primed by two different sample carts in the handoff note, varied quantity — and that’s the axis the bug lives on. Effort was equal. Coverage wasn’t.
• Two independent reviewers agreed on less than a quarter of what they found. 2 of 9 distinct findings were shared, and even one of those two got different severity ratings from each reviewer. A single review — of either kind — is a sample, not a ceiling.
• The review pass cost more than the code it reviewed. 122,605 tokens across both reviewers, against 98,856 to write the two files in the first place. Stacked across the full pipeline, verifying a two-function module cost 221,461 tokens — a number that only looks reasonable next to what a one-cent receipt bug costs when nobody catches it.

References