Benchmarks

Measured September 2026.

These are our own measurements, on our own test set, with the method and sample sizes stated. The set is small, so read the numbers as a guide rather than a guarantee: results vary with the codebase and the change being reviewed.

Results by level

• Budget

  $0.19 per review

  Labelled defects found

  72% 43 of 60

  Findings that were real

  99% 122 of 123

  Real defects per review

  5.5 3.0 of them P0/P1

  11 pull requests, 2 reviews each (22), September 2026. Measured as Budget’s core review, before checks added since.

• Strong Default

  $0.45 per review

  Labelled defects found

  78% 47 of 60

  Findings that were real

  98% 172 of 175

  Real defects per review

  6.9 3.7 of them P0/P1

  Same 11 pull requests, 2 reviews each (22), September 2026. Measured as Strong’s core review, before checks added since.

• Ultra

  $1.49 per review

  Not published yet. Ultra is Strong plus the Expert security and Expert logic add-ons. It has not been measured end to end, and our add-on measurements so far are too small, and partly on test cases we don’t publish from, to quote here.

  We will add it once it is measured on our own set.

Small sample: read these as a guide, not a guarantee. Results vary with the codebase and the change. Budget and Strong were measured on the same set, so their numbers can be compared with each other. The cost shown is the price you pay per review.

Method

• The set. 11 real pull requests from production repositories, with 30 defects labelled by hand. Budget and Strong each reviewed each pull request twice (22 reviews), replayed at the original commit.
• Labelled defects found counts, over all 22 reviews, how often a review reported a labelled defect (2 reviews × 30 defects = 60 chances).
• Findings that were real: a finding that matched a labelled defect counts as real; every other finding was checked against the code it points at and judged a real problem or noise.
• Real defects per review: real findings from every variant we tested were pooled and de-duplicated into 377 unique defects on the 11 pull requests; this is how many of them one review found, on average. P0/P1 are the two most severe grades.

What we left out

• Numbers from public benchmark sets built by third parties. We publish results on our own set only.
• Comparisons with other products.