# Allocating review attention to code changes: report
Chapter 2.2 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes. Table n of this report is figure n of the chapter.
## Identification
- Title: Allocating review attention to code changes
- Chapter: 2.2 (Part II, Scarce attention; Case studies)
- Question: Which agent-generated code changes should get a specialist's limited review time, and how much does a good score on a test set say about real work?
- Dek: On patches written so that every one passes the required tests, the models rank defective changes well above acceptable ones while diff size does no better than chance, but on real agent patches the same models do only modestly better than chance.
- Grain: records
- Status: draft
- Published: 2026-10-30
- Updated: not revised
- Data sources:
- Constructed patches. Patches on four small projects written for this booklet, with hidden issue tests; weights: unweighted; evenly split by construction
- SWE-rebench. Issues in sampled repositories; agent patches from the Nebius OpenHands trajectories; weights: sample_weight restores the outcome mix of the patches
- SWE-rebench OpenHands trajectories. Agent patches and the resolved outcome; weights: sample_weight restores the outcome mix of the patches
- Canonical URL: https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes
- Report: https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes/report
- Markdown: https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes/report.md
- BibTeX key: singh2026allocatingreviewattentiontocodechanges
```bibtex
@misc{singh2026allocatingreviewattentiontocodechanges,
author = {Singh, Vishal},
title = {Allocating review attention to code changes},
year = {2026},
url = {https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes},
note = {In Decision AI. Published 30 October 2026.}
}
```
## Principal estimates
The facts the dek and the opening prose cite, in order of appearance.
| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 10 | | | unweighted; evenly split by construction | Constructed patches | found10_luna |
| defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 9 | | | unweighted; evenly split by construction | Constructed patches | found10_jev |
| defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 5 | | | unweighted; evenly split by construction | Constructed patches | found10_size |
## Findings
An AI agent opens more code changes in a week than a specialist can read. In the workflow this chapter studies, every change has passed the required tests, which always run, and no model's opinion replaces them. That is true of the patches written for this booklet. The real benchmark patches later in the chapter were not filtered this way. The question is which changes get the specialist's limited time. This chapter ranks changes by model judgment and by simple rules, then asks how many problems each ranking finds.
On patches written for this booklet, where the flaw is there to be read in the diff, the models find defects far faster than sorting by size: Luna's ten riskiest hold 10 defects, Jev's 9 and the ten largest 5. On real agent patches the same models show no clear advantage over size. So a good score on a tidy test set is weak evidence about your own queue. The written set's mix is unlike any real queue, and the real set's label is a benchmark outcome, not a count of bugs that would ship.
### A test set where the answer is known
Four small projects were written for this booklet, with 20 issues and two patches for each: one acceptable and one with a real flaw, such as a removed permission check or an off-by-one error. All 40 patches pass the required tests. A hidden test, never shown to any model, settles which is which.
These patches are written for teaching and split evenly, so far more of them are defective than in a real queue. Results from this set are not pooled with real patches.
### Models find the flaws sooner than size does
Each model read the issue, the diff and the guidance, and answered: would this patch fail tests written to check that this issue is fixed? Sort by that answer and review from the top.
Table 1. Ranking by the models finds the defects much sooner than ranking by diff size
Sorting by size finds defects no faster than chance, because a small change can remove a permission check. A keyword rule looking for words like "auth" or "security" in added lines matches 0 of these patches, a quirk of how the set was written and not a finding about real code. Within each issue, Jev rates the defective patch riskier in 18.5 of 20 pairs and Luna in 18, so these rankings do more than react to which issue is harder.
Table 2. One question does most of the work, and equal weights dilute it
Splitting the judgment into smaller questions is a popular idea, and research on simple scoring rules gives it real support [1]. Here the result depends on the weights. For Jev, the single question "does it address the issue?" scores 0.88, while an equal-weight score across all the smaller questions scores 0.66, because questions about security or public interfaces say little about these particular flaws. Smaller questions still help a reviewer see why a patch looks risky, and a different mix of defects could favor them.
### Spending a limited review budget
The panel below turns a ranking into a week's budget. Suppose a defect that ships costs $2,000, a reviewer costs $150 an hour and finds a defect with probability 0.7. All three are assumptions, and a team would estimate them from incident records and review logs.
Double the hours. Which policy gains most from the extra time, and what happens to the gap between the models and random order?
The panel also includes a ranking by risk per review minute. It barely differs here because these patches take similar time to review; it would matter more in a queue with some very large changes.
### Real agent patches are much harder
The written patches have flaws a careful reader can see. Real failures are often subtler. To test that, the same models read 1,941 patches that one coding agent wrote for 259 real open-source issues. 44% of them failed the benchmark's tests for their issue. Another 59 runs ended with no change at all, usually when the agent ran out of steps. Those are left out, because there is nothing to review, and a check for an empty diff catches every one of them.
Table 3. On real agent patches, neither the models nor the rules clearly beat sorting by size
Across all patches, Jev scores 0.60, Luna 0.62 and largest diff first scores 0.60, so Jev's lead over size is +0.01 (95% CI −0.03 to 0.05). Comparing only patches for the same issue, in the 215 issues that have both outcomes, Jev scores 0.58, Luna 0.56 and size 0.52, and Jev's lead is +0.06 (95% CI −0.04 to 0.16). Both ranges include zero. That drop is the lesson to keep: a score on a tidy test set says little about performance on your real work.
This label cannot price a review budget. "Unresolved" also includes patches that existing tests would have caught and patches that do not apply, so it is not the population a specialist would see.
All the visible tests pass, but the issue's acceptance criteria look incomplete. What further evidence should the system ask for? Then look at the two sets of results: what would you test on your own code before giving a model's ranking any say over a specialist's time?
Show the budget panel first and ask students which policy they would buy. Then show the real-patch chart and ask what they would now want to see from a vendor. A semantic judgment never justifies skipping a required test, and the class should be able to say why.
### What these data cannot show
The written patches were made for this booklet with a few kinds of defect, so their scores are not an estimate for any real repository. The real patches come from one agent on one set of issues, their label is not the same as a defect that would ship, and the file-ownership map is an unreviewed draft. These open-source repositories may appear in the models' training data, which comparing patches for the same issue reduces and does not remove. No ranking here was used by real reviewers, whose detection rate and time are assumed, so nothing shows that a team would catch more defects in practice.
## Tables
### Table 1. Ranking by the models finds the defects much sooner than ranking by diff size
Defective patches among the first reviews in each order, constructed patches, if each review reveals the defect
| found | policy | reviews |
| ---: | --- | ---: |
| 1 | Jev 1.14.0 | 1 |
| 2 | Jev 1.14.0 | 2 |
| 3 | Jev 1.14.0 | 3 |
| 4 | Jev 1.14.0 | 4 |
| 5 | Jev 1.14.0 | 5 |
| 5 | Jev 1.14.0 | 6 |
| 6 | Jev 1.14.0 | 7 |
| 7 | Jev 1.14.0 | 8 |
| 8 | Jev 1.14.0 | 9 |
| 9 | Jev 1.14.0 | 10 |
| 10 | Jev 1.14.0 | 11 |
| 11 | Jev 1.14.0 | 12 |
| 12 | Jev 1.14.0 | 13 |
| 12 | Jev 1.14.0 | 14 |
| 12 | Jev 1.14.0 | 15 |
| 12 | Jev 1.14.0 | 16 |
| 13 | Jev 1.14.0 | 17 |
| 13 | Jev 1.14.0 | 18 |
| 14 | Jev 1.14.0 | 19 |
| 14 | Jev 1.14.0 | 20 |
| 14 | Jev 1.14.0 | 21 |
| 14 | Jev 1.14.0 | 22 |
| 14 | Jev 1.14.0 | 23 |
| 15 | Jev 1.14.0 | 24 |
| 16 | Jev 1.14.0 | 25 |
| 16 | Jev 1.14.0 | 26 |
| 17 | Jev 1.14.0 | 27 |
| 17 | Jev 1.14.0 | 28 |
| 17 | Jev 1.14.0 | 29 |
| 17 | Jev 1.14.0 | 30 |
| 17 | Jev 1.14.0 | 31 |
| 18 | Jev 1.14.0 | 32 |
| 18 | Jev 1.14.0 | 33 |
| 19 | Jev 1.14.0 | 34 |
| 19 | Jev 1.14.0 | 35 |
| 19 | Jev 1.14.0 | 36 |
| 19 | Jev 1.14.0 | 37 |
| 20 | Jev 1.14.0 | 38 |
| 20 | Jev 1.14.0 | 39 |
| 20 | Jev 1.14.0 | 40 |
| 1 | OpenAI Luna | 1 |
| 2 | OpenAI Luna | 2 |
| 3 | OpenAI Luna | 3 |
| 4 | OpenAI Luna | 4 |
| 5 | OpenAI Luna | 5 |
| 6 | OpenAI Luna | 6 |
| 7 | OpenAI Luna | 7 |
| 8 | OpenAI Luna | 8 |
| 9 | OpenAI Luna | 9 |
| 10 | OpenAI Luna | 10 |
| 11 | OpenAI Luna | 11 |
| 12 | OpenAI Luna | 12 |
| 13 | OpenAI Luna | 13 |
| 14 | OpenAI Luna | 14 |
| 14 | OpenAI Luna | 15 |
| 15 | OpenAI Luna | 16 |
| 16 | OpenAI Luna | 17 |
| 16 | OpenAI Luna | 18 |
| 16 | OpenAI Luna | 19 |
| 17 | OpenAI Luna | 20 |
| 17 | OpenAI Luna | 21 |
| 17 | OpenAI Luna | 22 |
| 17 | OpenAI Luna | 23 |
| 18 | OpenAI Luna | 24 |
| 18 | OpenAI Luna | 25 |
| 18 | OpenAI Luna | 26 |
| 18 | OpenAI Luna | 27 |
| 18 | OpenAI Luna | 28 |
| 18 | OpenAI Luna | 29 |
| 18 | OpenAI Luna | 30 |
| 18 | OpenAI Luna | 31 |
| 18 | OpenAI Luna | 32 |
| 18 | OpenAI Luna | 33 |
| 18 | OpenAI Luna | 34 |
| 18 | OpenAI Luna | 35 |
| 19 | OpenAI Luna | 36 |
| 20 | OpenAI Luna | 37 |
| 20 | OpenAI Luna | 38 |
| 20 | OpenAI Luna | 39 |
| 20 | OpenAI Luna | 40 |
| 1 | Claude Haiku adapter | 1 |
| 2 | Claude Haiku adapter | 2 |
| 3 | Claude Haiku adapter | 3 |
| 4 | Claude Haiku adapter | 4 |
| 5 | Claude Haiku adapter | 5 |
| 6 | Claude Haiku adapter | 6 |
| 7 | Claude Haiku adapter | 7 |
| 8 | Claude Haiku adapter | 8 |
| 9 | Claude Haiku adapter | 9 |
| 10 | Claude Haiku adapter | 10 |
| 10 | Claude Haiku adapter | 11 |
| 10 | Claude Haiku adapter | 12 |
| 11 | Claude Haiku adapter | 13 |
| 12 | Claude Haiku adapter | 14 |
| 12 | Claude Haiku adapter | 15 |
| 13 | Claude Haiku adapter | 16 |
| 14 | Claude Haiku adapter | 17 |
| 14 | Claude Haiku adapter | 18 |
| 14 | Claude Haiku adapter | 19 |
| 15 | Claude Haiku adapter | 20 |
| 15 | Claude Haiku adapter | 21 |
| 15 | Claude Haiku adapter | 22 |
| 15 | Claude Haiku adapter | 23 |
| 15 | Claude Haiku adapter | 24 |
| 15 | Claude Haiku adapter | 25 |
| 16 | Claude Haiku adapter | 26 |
| 16 | Claude Haiku adapter | 27 |
| 17 | Claude Haiku adapter | 28 |
| 18 | Claude Haiku adapter | 29 |
| 18 | Claude Haiku adapter | 30 |
| 18 | Claude Haiku adapter | 31 |
| 18 | Claude Haiku adapter | 32 |
| 18 | Claude Haiku adapter | 33 |
| 18 | Claude Haiku adapter | 34 |
| 18 | Claude Haiku adapter | 35 |
| 18 | Claude Haiku adapter | 36 |
| 19 | Claude Haiku adapter | 37 |
| 19 | Claude Haiku adapter | 38 |
| 20 | Claude Haiku adapter | 39 |
| 20 | Claude Haiku adapter | 40 |
| 0 | Largest diff first | 1 |
| 0 | Largest diff first | 2 |
| 1 | Largest diff first | 3 |
| 2 | Largest diff first | 4 |
| 2 | Largest diff first | 5 |
| 2 | Largest diff first | 6 |
| 3 | Largest diff first | 7 |
| 3 | Largest diff first | 8 |
| 4 | Largest diff first | 9 |
| 5 | Largest diff first | 10 |
| 5 | Largest diff first | 11 |
| 6 | Largest diff first | 12 |
| 7 | Largest diff first | 13 |
| 7 | Largest diff first | 14 |
| 8 | Largest diff first | 15 |
| 8 | Largest diff first | 16 |
| 8 | Largest diff first | 17 |
| 8 | Largest diff first | 18 |
| 9 | Largest diff first | 19 |
| 9 | Largest diff first | 20 |
| 10 | Largest diff first | 21 |
| 10 | Largest diff first | 22 |
| 10 | Largest diff first | 23 |
| 11 | Largest diff first | 24 |
| 12 | Largest diff first | 25 |
| 12 | Largest diff first | 26 |
| 12 | Largest diff first | 27 |
| 13 | Largest diff first | 28 |
| 13 | Largest diff first | 29 |
| 14 | Largest diff first | 30 |
| 15 | Largest diff first | 31 |
| 15 | Largest diff first | 32 |
| 16 | Largest diff first | 33 |
| 17 | Largest diff first | 34 |
| 17 | Largest diff first | 35 |
| 18 | Largest diff first | 36 |
| 19 | Largest diff first | 37 |
| 19 | Largest diff first | 38 |
| 20 | Largest diff first | 39 |
| 20 | Largest diff first | 40 |
| 0.500 | Random order (average) | 1 |
| 1 | Random order (average) | 2 |
| 1.50 | Random order (average) | 3 |
| 2 | Random order (average) | 4 |
| 2.50 | Random order (average) | 5 |
| 3 | Random order (average) | 6 |
| 3.50 | Random order (average) | 7 |
| 4 | Random order (average) | 8 |
| 4.50 | Random order (average) | 9 |
| 5 | Random order (average) | 10 |
| 5.50 | Random order (average) | 11 |
| 6 | Random order (average) | 12 |
| 6.50 | Random order (average) | 13 |
| 7 | Random order (average) | 14 |
| 7.50 | Random order (average) | 15 |
| 8 | Random order (average) | 16 |
| 8.50 | Random order (average) | 17 |
| 9 | Random order (average) | 18 |
| 9.50 | Random order (average) | 19 |
| 10 | Random order (average) | 20 |
| 10.50 | Random order (average) | 21 |
| 11 | Random order (average) | 22 |
| 11.50 | Random order (average) | 23 |
| 12 | Random order (average) | 24 |
| 12.50 | Random order (average) | 25 |
| 13 | Random order (average) | 26 |
| 13.50 | Random order (average) | 27 |
| 14 | Random order (average) | 28 |
| 14.50 | Random order (average) | 29 |
| 15 | Random order (average) | 30 |
| 15.50 | Random order (average) | 31 |
| 16 | Random order (average) | 32 |
| 16.50 | Random order (average) | 33 |
| 17 | Random order (average) | 34 |
| 17.50 | Random order (average) | 35 |
| 18 | Random order (average) | 36 |
| 18.50 | Random order (average) | 37 |
| 19 | Random order (average) | 38 |
| 19.50 | Random order (average) | 39 |
| 20 | Random order (average) | 40 |
Note: AUC on this set: Luna 0.89, Jev 0.79, Haiku adapter 0.78, largest diff first 0.45.
Source: Constructed patches written for this booklet, October 2026; 40 patches on 20 issues; unweighted
Data: https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/defects-found-by-review.json
### Table 2. One question does most of the work, and equal weights dilute it
AUC of each typed question for finding defective patches, constructed patches, by engine
| auc | model | signal |
| ---: | --- | --- |
| 0.785 | Jev 1.14.0 | Whole question: would it fail? |
| 0.662 | Jev 1.14.0 | Equal weight on the six questions |
| 0.877 | Jev 1.14.0 | Addresses the issue |
| 0.535 | Jev 1.14.0 | Changes a public interface |
| 0.499 | Jev 1.14.0 | Security sensitive |
| 0.666 | Jev 1.14.0 | Unrelated changes |
| 0.581 | Jev 1.14.0 | Tests added |
| 0.797 | Jev 1.14.0 | Too little context to judge |
| 0.887 | OpenAI Luna | Whole question: would it fail? |
| 0.681 | OpenAI Luna | Equal weight on the six questions |
| 0.874 | OpenAI Luna | Addresses the issue |
| 0.583 | OpenAI Luna | Changes a public interface |
| 0.502 | OpenAI Luna | Security sensitive |
| 0.586 | OpenAI Luna | Unrelated changes |
| 0.512 | OpenAI Luna | Tests added |
| 0.764 | OpenAI Luna | Too little context to judge |
Source: Constructed patches written for this booklet, October 2026; 40 patches on 20 issues; unweighted
Data: https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/which-question-helps.json
### Table 3. On real agent patches, neither the models nor the rules clearly beat sorting by size
AUC for the benchmark's unresolved label, across all patches and within the same issue, with 95% intervals
| across | hi1 | hi2 | lo1 | lo2 | n1 | n2 | policy | within |
| ---: | ---: | ---: | ---: | ---: | ---: | ---: | --- | ---: |
| 0.604 | 0.642 | 0.646 | 0.559 | 0.506 | 1941 | 1941 | Jev 1.14.0, whole question | 0.580 |
| 0.618 | 0.665 | 0.642 | 0.574 | 0.490 | 1941 | 1941 | OpenAI Luna, whole question | 0.557 |
| 0.608 | 0.651 | 0.624 | 0.567 | 0.502 | 1941 | 1941 | Claude Haiku adapter, whole question | 0.562 |
| 0.572 | 0.614 | 0.678 | 0.530 | 0.524 | 1941 | 1941 | Jev 1.14.0, equal weight | 0.595 |
| 0.542 | 0.580 | 0.667 | 0.498 | 0.503 | 1941 | 1941 | Jev 1.14.0, fitted weights | 0.578 |
| 0.597 | 0.648 | 0.608 | 0.554 | 0.442 | 1941 | 1941 | Largest diff first | 0.517 |
| 0.543 | 0.594 | 0.601 | 0.494 | 0.444 | 1941 | 1941 | File-ownership rule | 0.512 |
| 0.575 | 0.623 | 0.608 | 0.531 | 0.438 | 1941 | 1941 | Keyword rule | 0.515 |
Source: SWE-rebench and Nebius OpenHands trajectories, issues created by 2025; 2,000 patches, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/same-issue-is-harder.json
## All estimates
Every fact in facts.json (33), sorted by id.
| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| auc_across_diff_jev_size | +0.01 | difference in AUC | jev minus largest diff first | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI −0.03 to 0.05 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_across_diff_luna_size | +0.02 | difference in AUC | luna minus largest diff first | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI −0.01 to 0.06 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_across_jev | 0.60 | area under the ROC curve | unresolved ranked above resolved, patches for different issues | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.56 to 0.64 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_across_luna | 0.62 | area under the ROC curve | unresolved ranked above resolved, patches for different issues | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.57 to 0.66 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_across_size | 0.60 | area under the ROC curve | unresolved ranked above resolved, patches for different issues | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.55 to 0.65 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_addr_jev | 0.88 | area under the ROC curve | the addresses-the-issue question alone | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_con_adapter | 0.78 | area under the ROC curve | ranks defective patches above acceptable ones | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 95% CI 0.64 to 0.89 | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_con_jev | 0.79 | area under the ROC curve | ranks defective patches above acceptable ones | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 95% CI 0.69 to 0.89 | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_con_luna | 0.89 | area under the ROC curve | ranks defective patches above acceptable ones | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 95% CI 0.77 to 0.99 | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_con_size | 0.45 | area under the ROC curve | ranks defective patches above acceptable ones | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 95% CI 0.33 to 0.55 | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_dawes_jev | 0.66 | area under the ROC curve | equal weight on six direction-fixed questions | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | 40 patches | unweighted; evenly split by construction | Constructed patches | |
| auc_within_diff_jev_size | +0.06 | difference in AUC | jev minus largest diff first | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI −0.04 to 0.16 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_within_diff_luna_size | +0.04 | difference in AUC | luna minus largest diff first | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI −0.05 to 0.13 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_within_jev | 0.58 | area under the ROC curve | unresolved ranked above resolved, patches for the same issue | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.51 to 0.65 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_within_luna | 0.56 | area under the ROC curve | unresolved ranked above resolved, patches for the same issue | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.49 to 0.64 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| auc_within_size | 0.52 | area under the ROC curve | unresolved ranked above resolved, patches for the same issue | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.44 to 0.61 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| detect_rate | 0.7 | share of defects a reviewer finds | assumed reviewer detection rate | a weekly review budget (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Constructed patches | |
| found10_jev | 9 | defective patches of 20 | defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| found10_luna | 10 | defective patches of 20 | defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| found10_size | 5 | defective patches of 20 | defective patches among the ten ranked riskiest | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| hour_cost | $150 | US dollars per reviewer hour | assumed cost | a weekly review budget (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Constructed patches | |
| keyword_hits | 0 | patches of 40 | patches whose added lines match auth, security or api | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| miss_cost | $2,000 | US dollars per defect that ships | assumed cost | a weekly review budget (assumed inputs) | assumed inputs, October 2026 | | | not applicable; assumed inputs | Constructed patches | |
| n_con | 40 | patches | constructed patches | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| n_empty | 59 | agent runs | runs that ended with no diff, excluded | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| n_hist | 1,941 | patches | sampled agent patches | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| n_hist_issues | 259 | issues | issues in the sample | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| n_issues | 20 | issues | issues, each with one acceptable and one defective patch | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| n_mixed | 215 | issues | issues with both resolved and unresolved patches | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| pairs_jev | 18.5 | issues of 20 | defective patch rated riskier than the acceptable patch for the same issue; a tie is split evenly | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| pairs_luna | 18 | issues of 20 | defective patch rated riskier than the acceptable patch for the same issue; a tie is split evenly | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | | | unweighted; evenly split by construction | Constructed patches | |
| share_empty | 2.6% | percent of agent runs | runs that ended with no diff, weighted | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
| share_unresolved | 43.9% | percent of patches | benchmark outcome is unresolved | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | | | sample_weight restores the outcome mix of the patches | SWE-rebench | |
## Definitions
The chapter marks no glossary term.
## Methods
Two decision engines and a general language model read each patch with its issue and the repository's guidance and answered seven typed questions. Constructed patches and historical patches are reported separately and never pooled.
### The constructed patches
Four small projects were written for this booklet, with an issue, an acceptable patch and a defective patch for each issue. Every patch passes the project's required tests, and a hidden test settles whether it fixes the issue. The patches are evenly split by construction, and the keyword rule is a poor fit by construction, so neither says anything about real repositories.
The budget view uses the assumed reviewer costs and allows part of a patch at the budget's edge, which is an approximation to choosing whole patches.
### The historical patches
One agent's patches for real open-source issues. The label is the benchmark's unresolved outcome, which also counts patches that existing tests would have caught and patches that do not apply. So this set shows how well an outcome can be predicted, and it does not count defects that would reach production. These repositories and issues may appear in the models' training data; comparing patches for the same issue reduces that problem and does not remove it.
Models saw diffs without the agent's leftover scripts. The file-ownership map is a draft that was not reviewed by the repositories' maintainers.
### Intervals
Constructed intervals resample issues, keeping both patches together. Historical intervals resample issues within repositories, with fewer replicates because same-issue AUC is slow to compute. A missing answer is ranked at the sample mean, and refusals are counted in the chapter.
## What these data cannot show
The written patches were made for this booklet with a few kinds of defect, so their scores are not an estimate for any real repository. The real patches come from one agent on one set of issues, their label is not the same as a defect that would ship, and the file-ownership map is an unreviewed draft. These open-source repositories may appear in the models' training data, which comparing patches for the same issue reduces and does not remove. No ranking here was used by real reviewers, whose detection rate and time are assumed, so nothing shows that a team would catch more defects in practice.
## References
1. Dawes, Robyn M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34(7), 571-582. https://doi.org/10.1037/0003-066X.34.7.571
## Provenance
- Build script: decision_ai/chapters/allocating-review-attention-to-code-changes/build.py (sha256 9a26845e7af2cd3fee289e9630841a19c3c5f921d9c41cba96e0adc5f9873221)
- Built on: 2026-10-30
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0
Outputs:
| File | SHA-256 |
| --- | --- |
| facts.json | 101cdf86f86fe57545d87f5eac1eaeb4f36c590aa3360dc1876ba4f0f007dc4c |
| figures/defects-found-by-review.json | dfd3ed5cba0f31b34d0407d9fd00486cb9798136c11671a679cb4e6f1acc2c80 |
| figures/same-issue-is-harder.json | ab9e25f79f5342897a30c92d8619bf051fa69489d0efbd7e18cb576c81fdec62 |
| figures/which-question-helps.json | e2c337d5d02164f2a7c72749427dbae8ebe71e4a3ad4b564f85a225764fbad2c |
| meta.json | efa66d99b4dc878a54b8c4f5a3459e084b947f9039bf31a7f2183e2cfe670670 |
| public/data/allocating-review-attention-to-code-changes/budget.json | b7794c5b863c43e51f8978175b4680051349409ca49b5a67d57e4f36476253c9 |
| public/data/allocating-review-attention-to-code-changes/reveal.json | 8782bc8d18cf279067f365dd981c7681a78c4b6722b4f6f92e75f2d5c56053ea |
| refs.json | f02a9f2b53002b0cc1ba8e133d2caf0bd4e7b7f4593ca1534ead259450db78d3 |
## Machine formats
- Report (HTML): https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes/report
- Report (Markdown): https://vishalsingh.org/decisionai/allocating-review-attention-to-code-changes/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/facts.json
- Table 1, figure "defects-found-by-review" (JSON): https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/defects-found-by-review.json
- Table 2, figure "which-question-helps" (JSON): https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/which-question-helps.json
- Table 3, figure "same-issue-is-harder" (JSON): https://vishalsingh.org/decisionai/report/allocating-review-attention-to-code-changes/figures/same-issue-is-harder.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt