# Comparing decision systems: report

Chapter 4.1 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems. Table n of this report is figure n of the chapter.

## Identification

- Title: Comparing decision systems
- Chapter: 4.1 (Part IV, Deciding whether to adopt; Case studies)
- Question: Across the cases, when should a team write a rule, train a classifier, ask a decision endpoint or keep a person, and what should it ask before adopting a system?
- Dek: Across reviews, complaints and agent work, the choice between an endpoint and a trained classifier turns on how many labels a task needs and whether the probabilities are calibrated, and an agent's work improved most when a check it could not influence named what had failed.
- Grain: records
- Status: draft
- Published: 2026-10-30
- Updated: not revised
- Data sources:
  - Amazon Reviews 2023. Results reused from the two review chapters; weights: sample_weight restores the star mix
  - CFPB Consumer Complaint Database. Results reused from the complaint chapter; weights: sample_weight restores the product mix
  - SWE-rebench. Results reused from the code-review chapter; weights: sample_weight restores the outcome mix
  - Constructed patches. Results reused from the code-review chapter; weights: unweighted; evenly split by construction
  - Constructed agent tasks. Results reused from the agent chapter; weights: unweighted; the mix describes the test harness
- Canonical URL: https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems
- Report: https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems/report
- Markdown: https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems/report.md
- BibTeX key: singh2026comparingandauditingdecisionsystems

```bibtex
@misc{singh2026comparingandauditingdecisionsystems,
  author = {Singh, Vishal},
  title = {Comparing decision systems},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems},
  note = {In Decision AI. Published 30 October 2026.}
}
```

## Principal estimates

The facts the dek and the opening prose cite, in order of appearance.

| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| mean net value over refits on 1,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $4,466 |   | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | lc_net_1000 |
| mean net value over refits on 20,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $5,047 |   | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | lc_net_20000 |
| net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | $5,076 | 95% CI $5,004 to $5,144 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | net_jev |
| mean top-tenth capture over refits on 1,000 labeled complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 58.8% |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | lc_capture_1000 |
| share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 58.2% | 95% CI 53.0% to 63.4% |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | capture_jev |
| share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 73.1% | 95% CI 68.4% to 77.9% |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | capture_tfidf |

## Findings

Trained on a thousand labeled reviews, a classifier's contact policy earns $4,500 per thousand reviews at the assumed costs, and it takes the full labeled sample to reach $5,000, about what raw Jev earns with no labels ($5,100). On money back the order reverses: a classifier with about a thousand past complaints catches 59% of those cases in its top tenth, level with Jev's 58%, and with the full sample it reaches 73%.

So how much local data a task needs differs widely, and it is an important practical difference between an endpoint and a classifier. A good ranking is not enough when the action rule takes probabilities at face value, and a score on a tidy test set says little about real work. These results come from a handful of cases, with stand-in labels and assumed costs, so treat them as questions to ask of your own data.

### What local labels change

On reading a review, the endpoint needs no training and the classifier needs a large labeled sample to catch up. On predicting what a company will do, the classifier learns from records a desk already holds and passes the endpoint with a modest sample. The reason is a matter of interpretation: reading can draw on what a language model already knows, while a company's habits sit in its own records. These two cases are consistent with that reading and cannot test it.

When both uses get the same labels, the comparison sharpens. On the contact decision, two hundred labels give $5,300 spent recalibrating Jev and $3,300 spent fitting a classifier. On complaint routing, five hundred labels give expected costs of $3,600 and $4,800. So for reading tasks, a modest budget of labels went further checking an endpoint, in these two tests and with this one classifier. For forecasting what a company will do, it went further training on outcomes.

### A good ranking is not enough when the action rule takes probabilities at face value

The next chart puts each priced decision on one scale, comparing Jev's policy with the simple alternative in that case.

Table 1. Whether Jev's policy beats the simple alternative turns on calibrating its probabilities

In the review case, costs alone put the contact cutoff at 0.125, while the data put Jev's at 0.38 because its probabilities run high. Raw Jev earns $5,100 and the classifier $5,000, with no clear gap; recalibrated Jev earns $5,300. On complaints the raw comparison favors the classifier, and with both recalibrated the gap is −$195 (95% CI −$419 to $27) dollars per thousand. So a firm that applies a cost cutoff to an endpoint's raw probabilities may get less than its ranking score suggests, and a modest labeled sample is worth budgeting even if it never trains a model. A policy that only fills a fixed budget from the top of a ranking leans less on the probabilities themselves.

Call charges did not decide any of this. Jev costs about $0.05 per thousand complaints, against expected handling costs of $3,471 per thousand in the same assumed scenario. A full comparison would also count integration, checking and upkeep, which these figures leave out.

### A tidy test set says little about real work

Table 2. Jev's ranking score differs widely from one question to the next

Jev scores 0.79 on patches written with a visible flaw and 0.60 on real agent patches, where its lead over sorting by size is +0.01 (95% CI −0.03 to 0.05). Money back shows a related trap: a high score can come from something other than reading. Ranking complaints by their product's past rate alone gives 0.906, and within a product Jev scores 0.742 and the classifier 0.769.

### Evidence that names the problem beats a request to look again

An agent's first finish met every checked requirement in 88% of runs. Sending work back on Jev's judgment reached 90% and sending everything back 90%. A check the agent could not influence, naming the failed requirement, reached 94%. That check is the grader itself, an oracle-style comparator for the checked requirements, and the tasks were written for this booklet. The five automated policies were run; a person was not. The pattern matches what the same chapter found on simpler ground: after a tool reports a failure, a model chooses to retry 97% of the time. When the evidence says what happened, the next step is easy.

### Where to start

(Interactive guide: yes-or-no questions lead to a starting recommendation. The same routes are in the table that follows.)

The guide follows the routes in this table, each drawn from one case.

| Your situation | Where to start | Where it comes from |
| --- | --- | --- |
| An exact test can check the answer | The rule or the test, and report what failed | Chapter 3.1, Deciding when an agent is finished (https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished) |
| Predicting a later outcome, with many past examples | Evaluate a classifier trained on your outcomes | Chapter 2.1, Resolving and routing customer complaints (https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints) |
| Predicting a later outcome, with few past examples | Record outcomes first; use an endpoint only as a stopgap | Chapter 2.1, Resolving and routing customer complaints (https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints) |
| A reader could not tell from the evidence whether the answer is right | A person, or a check the system cannot influence | Chapter 3.1, Deciding when an agent is finished (https://vishalsingh.org/decisionai/deciding-when-an-agent-is-finished) |
| Reading the text, with thousands of labels of your own | Compare an endpoint and a classifier as complete policies | Chapter 1.3, From probabilities to actions (https://vishalsingh.org/decisionai/from-probabilities-to-actions) |
| Reading the text, with few labels | Try an endpoint, and spend a few hundred labels calibrating it before acting | Chapter 1.2, From evidence to typed questions (https://vishalsingh.org/decisionai/from-evidence-to-typed-questions) |

These are starting points drawn from a few cases and not rules. A team's own costs and records can reverse any of them.

### Would you deploy this? The complaint desk

Take the desk's routing decision and put the evidence in one place. The choices are to deploy, run a limited pilot, measure further or keep the current process.

| Evidence | What the case shows | What is still missing |
| --- | --- | --- |
| Judgment | Agreement with consumers' choices of 82% for Jev and 82% for the classifier | How often the consumer's choice is itself wrong |
| Probabilities | Neither model's stated probabilities can be taken at face value | Calibration on the desk's own complaints |
| Policy | Recalibrated, $3,471 for Jev and $3,666 for the classifier per thousand complaints, with no clear gap | The desk's real costs of delay and triage |
| Attention | Ranking review by expected loss per minute changes cost by −425 (95% CI −639 to −198) dollars per thousand, under assumed review times | Measured handling time, and reviewer accuracy on short complaints |
| Failures | Largest disagreements run between credit reporting and debt collection | What a misroute costs in each direction |
| Observed effect | None; no policy here ran on a desk | A randomized pilot |

<Predict id="ch-adopt" options="Deploy|Run a limited pilot|Measure further|Keep the current process">

Given only this evidence, what would you recommend to the desk? Choose before reading on.

</Predict>

This evidence does not justify full deployment. It justifies a local evaluation of whichever model the desk can run most cheaply, with its probabilities recalibrated on a labeled sample of the desk's own complaints, because neither shows a clear lead. Start in shadow, recording the model's picks while the current process keeps routing. Changing live routing also needs a measured baseline, a limited set of complaints and a way back. For the money-back question, closed-case records make a classifier a strong baseline to evaluate, once the desk has checked that outcomes arrive in time, that old records resemble today's complaints and that money back is what it wants to prioritize.

Adoption also changes what the desk can see. Misroutes that bounce back get recorded, and ones that get handled without complaint do not, so a small random sample of automatically routed complaints, reviewed on a regular schedule, is what would show the policy going stale.

### Questions to ask before adopting a system

Each case suggests a question for a vendor or an internal team.

| Question | Why it matters here |
| --- | --- |
| Was it tested on work like yours, or on a tidy test set? | The same engine scored far lower on real code changes than on written ones |
| What is the simplest alternative, and has it been run on the same records? | A classifier or sorting by size was hard to beat, and the classifier's result depended on how it was fit |
| What does the label actually measure? | Stars, consumer choices and company responses are all stand-ins |
| Do the probabilities mean what they say, near the cutoff you will use? | Recalibrating changed the answer in two of the priced decisions |
| What happens to the answers if the question changes? | Rewording one question changed 2.5% of Jev's answers |
| Who owns the question, the costs and the cutoff, and what triggers a recheck? | The engine, the costs and the population can each change on their own |
| Has anyone run this policy, or only scored it? | Only the agent case ran its policies, on tasks written for this booklet |

<Discuss>

Pick one decision your organization might hand to a model. Walk it through the guide. Where does it land, how many labels would it take to find out, and what would you need to measure before you trusted that answer?

</Discuss>

<InstructorNote>

Ask groups to choose a decision, walk it through the guide and defend the result against another group playing the vendor. Then ask each group to name the cheapest experiment that would change its mind.

</InstructorNote>

### What these data cannot show

This chapter adds no new evidence. The cases are few, each has its own label, population and assumed costs, and the pattern across them is an interpretation; another set of cases could look different. Only the agent policies were executed, on tasks written for this booklet, and no policy here was put into operation on real work. Jev stands in for one engine at one date. Consistency over time, changes between engine versions and sensitivity to signals that should not matter, which a full audit would cover, were not tested here.

## Tables

### Table 1. Whether Jev's policy beats the simple alternative turns on calibrating its probabilities

Jev's policy against the simple alternative, percent of that alternative's value or cost, with 95% intervals

| conf_high | conf_low | decision | gap |
| ---: | ---: | --- | ---: |
| 3.20 | −0.900 | Contacting reviewers, raw probabilities (against a classifier) | 1.20 |
| 8.20 | 4.50 | Contacting reviewers, Jev recalibrated (against a classifier) | 6.40 |
| −2.60 | −17.80 | Routing complaints, raw probabilities (against a classifier) | −10 |
| 11.40 | −0.700 | Routing complaints, both recalibrated (against a classifier) | 5.30 |
| 34.60 | 0 | Accepting an agent's finish (against a person checking all) | 17.50 |

Source: Case chapters of this booklet: coffee reviews and CFPB complaints, weighted; SWE-rebench patches, weighted; constructed patches and agent tasks, unweighted; engine calls October 2026; costs assumed

Data: https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/policy-gaps.json

### Table 2. Jev's ranking score differs widely from one question to the next

AUC by question, Jev and the trained classifier, with 95% intervals; no classifier was trained for the patch and agent cases

| auc | conf_high | conf_low | model | task |
| ---: | ---: | ---: | --- | --- |
| 0.980 | 0.982 | 0.978 | Jev | Is this reviewer unhappy? |
| 0.968 | 0.972 | 0.964 | Trained classifier | Is this reviewer unhappy? |
| 0.742 | 0.770 | 0.709 | Jev | Will this complaint end in money back, within a product? |
| 0.769 | 0.802 | 0.736 | Trained classifier | Will this complaint end in money back, within a product? |
| 0.785 | 0.890 | 0.690 | Jev | Would this written patch fail its hidden test? |
| 0.604 | 0.640 | 0.560 | Jev | Did this real agent patch fail its benchmark? |
| 0.713 | 0.830 | 0.550 | Jev | Is the agent's work really done? |

Source: Case chapters of this booklet: coffee reviews and CFPB complaints, weighted; SWE-rebench patches, weighted; constructed patches and agent tasks, unweighted; engine calls October 2026; costs assumed

Data: https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/same-model-three-questions.json

## All estimates

Every fact in facts.json (38), sorted by id.

| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| agree_jev | 81.9% | percent of complaints | top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 80.5% to 83.2% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| agree_tfidf | 82.4% | percent of complaints | top product equals the consumer-selected product | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 81.2% to 83.7% | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| auc_across_diff_jev_size | +0.01 | difference in AUC | jev minus largest diff first | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI −0.03 to 0.05 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | From allocating-review-attention-to-code-changes. |
| auc_done_jev | 0.71 | area under the ROC curve | P(done) against the grader's verdict on finish attempts | 200 finish attempts by one agent on 40 tasks written for this booklet | agent runs and engine calls, October 2026 | 95% CI 0.55 to 0.83 | 200 finish attempts | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| auc_money_jev | 0.895 | area under the ROC curve | P(money back) against closed with monetary relief | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.881 to 0.908 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| auc_money_product_only | 0.906 | area under the ROC curve | ranking complaints by their product's relief rate in the training years | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| auc_money_tfidf | 0.936 | area under the ROC curve | P(money back) against closed with monetary relief | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.928 to 0.945 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| auc_patch_real | 0.60 | area under the ROC curve | unresolved ranked above resolved, patches for different issues | 1,941 agent patches for 259 issues in 20 repositories | issues created by March 2025; engines called October 2026 | 95% CI 0.56 to 0.64 | 1,941 patches | sample_weight restores the outcome mix of the patches | SWE-rebench | From allocating-review-attention-to-code-changes. |
| auc_patch_written | 0.79 | area under the ROC curve | ranks defective patches above acceptable ones | 40 patches written for this booklet | patches written October 2026; engines called October 2026 | 95% CI 0.69 to 0.89 | 40 patches | unweighted; evenly split by construction | Constructed patches | From allocating-review-attention-to-code-changes. |
| auc_reviews_jev | 0.980 | area under the ROC curve | weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.978 to 0.982 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-evidence-to-typed-questions. |
| auc_within_jev | 0.742 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.709 to 0.770 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| auc_within_tfidf | 0.769 | area under the ROC curve | mean across products of the AUC within each product that has both outcomes | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 0.736 to 0.802 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| capture_jev | 58.2% | percent of complaints closed with monetary relief | share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 53.0% to 63.4% |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| capture_tfidf | 73.1% | percent of complaints closed with monetary relief | share caught by fast-tracking the tenth of complaints with the highest P(money back) | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI 68.4% to 77.9% |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| cost_complaints | $0.05 | US dollars per 1,000 complaints | mean billed cost per call, list price | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| cut_data | 0.38 | probability of dissatisfaction | median of five cross-fit cost-minimizing cutoffs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Range across folds 0.35 to 0.38. |
| cut_theory | 0.125 | probability of dissatisfaction | cutoff c / (rV) | the coffee roaster's contact decision (assumed inputs) | assumed inputs, October 2026 |   |   | not applicable; assumed inputs | Amazon Reviews 2023 | From from-probabilities-to-actions. |
| lc_capture_1000 | 58.8% | percent of complaints closed with monetary relief | mean top-tenth capture over refits on 1,000 labeled complaints | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 |   | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| lc_net_1000 | $4,466 | US dollars per 1,000 reviews | mean net value over refits on 1,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-evidence-to-typed-questions. |
| lc_net_20000 | $5,047 | US dollars per 1,000 reviews | mean net value over refits on 20,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 |   | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-evidence-to-typed-questions. |
| loss_complaints | $3,471 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,243 to $3,709 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. Assumed costs; measured error mix against the consumer's product. |
| loss_diff_jev_cal | −$195 | US dollars per 1,000 complaints | jev_cal minus the classifier with a cutoff | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −$419 to $27 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| loss_jev_cal | $3,471 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,243 to $3,709 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. Assumed costs; measured error mix against the consumer's product. |
| loss_tfidf_cal | $3,666 | US dollars per 1,000 complaints | expected misroute and triage cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI $3,466 to $3,885 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. Assumed costs; measured error mix against the consumer's product. |
| lv_clf_200 | $3,257 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Assumed costs. Both arms use the same labeled draws. |
| lv_cost_clf_500 | $4,792 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. Assumed costs. Both arms use the same labeled draws. |
| lv_cost_jev_500 | $3,613 | US dollars per thousand complaints | expected misroute and triage cost at the cutoff, mean across label draws | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints | complaints received in 2025; engines called October 2026 |   |   | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. Assumed costs. Both arms use the same labeled draws. |
| lv_jev_200 | $5,257 | US dollars per thousand reviews | net value at p*, mean across label draws | the coffee reviews, labels drawn from reviews of other products | reviews written 2006 to 2023; engines called October 2026 |   |   | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Assumed costs. Both arms use the same labeled draws. |
| met_accept | 87.5% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 81.0% to 93.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| met_check_named | 94.0% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 90.5% to 97.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| met_jev | 90.0% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 85.5% to 94.5% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| met_recheck | 89.5% | percent of runs | accepted final state meets every rule-checked criterion | 200 runs by one agent on 40 tasks written for this booklet, each policy run to the end of the task | agent runs and engine calls, October 2026 | 95% CI 85.0% to 94.0% | 200 runs | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| net_jev | $5,076 | US dollars per 1,000 reviews | net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $5,004 to $5,144 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Assumed costs; measured error mix. |
| net_jev_recal | $5,335 | US dollars per 1,000 reviews | net value per 1,000 reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $5,273 to $5,393 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Assumed costs; measured error mix. |
| net_tfidf | $5,015 | US dollars per 1,000 reviews | net value of contacts against contacting nobody, at the cutoff p* | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI $4,928 to $5,104 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-probabilities-to-actions. Assumed costs; measured error mix. |
| queue_diff_value_per_minute_cost | −425 | per 1,000 complaints | value_per_minute minus reviewing the least certain, cost | 10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters | complaints received in 2025; engines called October 2026 | 95% CI −639 to −198 | 10,000 complaints | weighted to the product mix of those complaints (sample_weight) | CFPB Consumer Complaint Database | From resolving-and-routing-customer-complaints. |
| retry_jev | 97.4% | percent of checkpoints | next action is retry after a transient failure | 38 checkpoints after an injected tool failure | agent runs and engine calls, October 2026 |   | 38 checkpoints | unweighted; the mix of tasks and faults describes the test harness | Constructed agent tasks | From deciding-when-an-agent-is-finished. |
| wording_change | 2.5% | percent of reviews | answer differs between the two versions of the question | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 1.4% to 3.9% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | From from-evidence-to-typed-questions. |

## Definitions

The chapter marks no glossary term.

## Methods

This chapter runs nothing new. Its numbers are copied by script from the other chapters' data files.

### Putting the priced decisions on one scale

The three decisions use different assumed costs, so each gap and its interval is divided by the simple alternative's dollar figure in the same case, with the sign set so that right means better for Jev. The denominators are treated as fixed, so these ranges are approximate. The percent scale helps compare direction and spread; it does not make the three assumed cost sets comparable.

## What these data cannot show

This chapter adds no new evidence. The cases are few, each has its own label, population and assumed costs, and the pattern across them is an interpretation; another set of cases could look different. Only the agent policies were executed, on tasks written for this booklet, and no policy here was put into operation on real work. Jev stands in for one engine at one date. Consistency over time, changes between engine versions and sensitivity to signals that should not matter, which a full audit would cover, were not tested here.

## References

The chapter cites no reference.

## Provenance

- Build script: decision_ai/chapters/comparing-and-auditing-decision-systems/build.py (sha256 756143a43f0dee2e10eb7f037f68ef8dd2e978818bf3dda7f2366edd2325f058)
- Built on: 2026-10-30
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0

Outputs:

| File | SHA-256 |
| --- | --- |
| facts.json | 2cfb55a8f2401a7aa72efc6a2b6d1640fa7810ec3fd37190c7dbc4f286c114cc |
| figures/policy-gaps.json | 03bcbc36dd93675885cb097ae587331a5f2b307df3898b86bebb39006076a018 |
| figures/same-model-three-questions.json | 0afa4141d1b4fb93f0fc5ddfe99aeace040d31f3dbcda3a53ed87a31a4babc34 |
| meta.json | 1d4dc1729ddbe586dc4909fd14343f12f3319e01cca1b54b0c52cd4824847050 |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |

## Machine formats

- Report (HTML): https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems/report
- Report (Markdown): https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/facts.json
- Table 1, figure "policy-gaps" (JSON): https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/policy-gaps.json
- Table 2, figure "same-model-three-questions" (JSON): https://vishalsingh.org/decisionai/report/comparing-and-auditing-decision-systems/figures/same-model-three-questions.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt