Chapter 4.1

Comparing decision systems

Across reviews, complaints and agent work, the choice between an endpoint and a trained classifier turns on how many labels a task needs and whether the probabilities are calibrated, and an agent's work improved most when a check it could not influence named what had failed.

Author: Vishal Singh
Affiliation: NYU Stern
Published: 30 October 2026

Full report: every estimate with its interval, and every figure as a table.

DraftFigures and numbers may change as the analysis is rebuilt.what is still being checked

Rewritten after the metrics review and the executed agent runs. The audit topics in the original plan were not run.

Trained on a thousand labeled reviews, a classifier's contact policy earns $4,500 per thousand reviews at the assumed costs, and it takes the full labeled sample to reach $5,000, about what raw Jev earns with no labels ($5,100). On money back the order reverses: a classifier with about a thousand past complaints catches 59% of those cases in its top tenth, level with Jev's 58%, and with the full sample it reaches 73%.

So how much local data a task needs differs widely, and it is an important practical difference between an endpoint and a classifier. A good ranking is not enough when the action rule takes probabilities at face value, and a score on a tidy test set says little about real work. These results come from a handful of cases, with stand-in labels and assumed costs, so treat them as questions to ask of your own data.

What local labels change

On reading a review, the endpoint needs no training and the classifier needs a large labeled sample to catch up. On predicting what a company will do, the classifier learns from records a desk already holds and passes the endpoint with a modest sample. The reason is a matter of interpretation: reading can draw on what a language model already knows, while a company's habits sit in its own records. These two cases are consistent with that reading and cannot test it.

When both uses get the same labels, the comparison sharpens. On the contact decision, two hundred labels give $5,300 spent recalibrating Jev and $3,300 spent fitting a classifier. On complaint routing, five hundred labels give expected costs of $3,600 and $4,800. So for reading tasks, a modest budget of labels went further checking an endpoint, in these two tests and with this one classifier. For forecasting what a company will do, it went further training on outcomes.

A good ranking is not enough when the action rule takes probabilities at face value

The next chart puts each priced decision on one scale, comparing Jev's policy with the simple alternative in that case.

Figure 1

Whether Jev's policy beats the simple alternative turns on calibrating its probabilities

Jev's policy against the simple alternative, percent of that alternative's value or cost, with 95% intervals

Show data

Data for Whether Jev's policy beats the simple alternative turns on calibrating its probabilities
conf_high	conf_low	decision	gap
3.20	−0.900	Contacting reviewers, raw probabilities (against a classifier)	1.20
8.20	4.50	Contacting reviewers, Jev recalibrated (against a classifier)	6.40
−2.60	−17.80	Routing complaints, raw probabilities (against a classifier)	−10
11.40	−0.700	Routing complaints, both recalibrated (against a classifier)	5.30
34.60	0	Accepting an agent's finish (against a person checking all)	17.50

Right of the line, Jev's policy does better than the simple alternative; left of it, worse. Compare each decision's raw and recalibrated rows. Recalibrating needs labeled records, which these figures do not charge for.Source: Case chapters of this booklet: coffee reviews and CFPB complaints, weighted; SWE-rebench patches, weighted; constructed patches and agent tasks, unweighted; engine calls October 2026; costs assumed

In the review case, costs alone put the contact cutoff at 0.125, while the data put Jev's at 0.38 because its probabilities run high. Raw Jev earns $5,100 and the classifier $5,000, with no clear gap; recalibrated Jev earns $5,300. On complaints the raw comparison favors the classifier, and with both recalibrated the gap is −$195 (95% CI −$419 to $27) dollars per thousand. So a firm that applies a cost cutoff to an endpoint's raw probabilities may get less than its ranking score suggests, and a modest labeled sample is worth budgeting even if it never trains a model. A policy that only fills a fixed budget from the top of a ranking leans less on the probabilities themselves.

Call charges did not decide any of this. Jev costs about $0.05 per thousand complaints, against expected handling costs of $3,471 per thousand in the same assumed scenario. A full comparison would also count integration, checking and upkeep, which these figures leave out.

A tidy test set says little about real work

Figure 2

Jev's ranking score differs widely from one question to the next

AUC by question, Jev and the trained classifier, with 95% intervals; no classifier was trained for the patch and agent cases

Jev
Trained classifier

Show data

Data for Jev's ranking score differs widely from one question to the next
auc	conf_high	conf_low	model	task
0.980	0.982	0.978	Jev	Is this reviewer unhappy?
0.968	0.972	0.964	Trained classifier	Is this reviewer unhappy?
0.742	0.770	0.709	Jev	Will this complaint end in money back, within a product?
0.769	0.802	0.736	Trained classifier	Will this complaint end in money back, within a product?
0.785	0.890	0.690	Jev	Would this written patch fail its hidden test?
0.604	0.640	0.560	Jev	Did this real agent patch fail its benchmark?
0.713	0.830	0.550	Jev	Is the agent's work really done?

Read down the rows. Each question has its own label, population and weighting, so these scores are not directly comparable. One reading is that they track how much of the answer is visible in the text, which these cases cannot test.Source: Case chapters of this booklet: coffee reviews and CFPB complaints, weighted; SWE-rebench patches, weighted; constructed patches and agent tasks, unweighted; engine calls October 2026; costs assumed

Jev scores 0.79 on patches written with a visible flaw and 0.60 on real agent patches, where its lead over sorting by size is +0.01 (95% CI −0.03 to 0.05). Money back shows a related trap: a high score can come from something other than reading. Ranking complaints by their product's past rate alone gives 0.906, and within a product Jev scores 0.742 and the classifier 0.769.

Evidence that names the problem beats a request to look again

An agent's first finish met every checked requirement in 88% of runs. Sending work back on Jev's judgment reached 90% and sending everything back 90%. A check the agent could not influence, naming the failed requirement, reached 94%. That check is the grader itself, an oracle-style comparator for the checked requirements, and the tasks were written for this booklet. The five automated policies were run; a person was not. The pattern matches what the same chapter found on simpler ground: after a tool reports a failure, a model chooses to retry 97% of the time. When the evidence says what happened, the next step is easy.

Where to start

Decision guide

Where should this decision start?

Can you write an exact test for the right answer?

Do the required tests pass? Did the tool report an error?

The guide follows the routes in this table, each drawn from one case.

Your situation	Where to start	Where it comes from
An exact test can check the answer	The rule or the test, and report what failed	Chapter 3.1, Deciding when an agent is finished
Predicting a later outcome, with many past examples	Evaluate a classifier trained on your outcomes	Chapter 2.1, Resolving and routing customer complaints
Predicting a later outcome, with few past examples	Record outcomes first; use an endpoint only as a stopgap	Chapter 2.1, Resolving and routing customer complaints
A reader could not tell from the evidence whether the answer is right	A person, or a check the system cannot influence	Chapter 3.1, Deciding when an agent is finished
Reading the text, with thousands of labels of your own	Compare an endpoint and a classifier as complete policies	Chapter 1.3, From probabilities to actions
Reading the text, with few labels	Try an endpoint, and spend a few hundred labels calibrating it before acting	Chapter 1.2, From evidence to typed questions

These are starting points drawn from a few cases and not rules. A team's own costs and records can reverse any of them.

Would you deploy this? The complaint desk

Take the desk's routing decision and put the evidence in one place. The choices are to deploy, run a limited pilot, measure further or keep the current process.

Evidence	What the case shows	What is still missing
Judgment	Agreement with consumers' choices of 82% for Jev and 82% for the classifier	How often the consumer's choice is itself wrong
Probabilities	Neither model's stated probabilities can be taken at face value	Calibration on the desk's own complaints
Policy	Recalibrated, $3,471 for Jev and $3,666 for the classifier per thousand complaints, with no clear gap	The desk's real costs of delay and triage
Attention	Ranking review by expected loss per minute changes cost by −425 (95% CI −639 to −198) dollars per thousand, under assumed review times	Measured handling time, and reviewer accuracy on short complaints
Failures	Largest disagreements run between credit reporting and debt collection	What a misroute costs in each direction
Observed effect	None; no policy here ran on a desk	A randomized pilot

This evidence does not justify full deployment. It justifies a local evaluation of whichever model the desk can run most cheaply, with its probabilities recalibrated on a labeled sample of the desk's own complaints, because neither shows a clear lead. Start in shadow, recording the model's picks while the current process keeps routing. Changing live routing also needs a measured baseline, a limited set of complaints and a way back. For the money-back question, closed-case records make a classifier a strong baseline to evaluate, once the desk has checked that outcomes arrive in time, that old records resemble today's complaints and that money back is what it wants to prioritize.

Adoption also changes what the desk can see. Misroutes that bounce back get recorded, and ones that get handled without complaint do not, so a small random sample of automatically routed complaints, reviewed on a regular schedule, is what would show the policy going stale.

Questions to ask before adopting a system

Each case suggests a question for a vendor or an internal team.

Question	Why it matters here
Was it tested on work like yours, or on a tidy test set?	The same engine scored far lower on real code changes than on written ones
What is the simplest alternative, and has it been run on the same records?	A classifier or sorting by size was hard to beat, and the classifier's result depended on how it was fit
What does the label actually measure?	Stars, consumer choices and company responses are all stand-ins
Do the probabilities mean what they say, near the cutoff you will use?	Recalibrating changed the answer in two of the priced decisions
What happens to the answers if the question changes?	Rewording one question changed 2.5% of Jev's answers
Who owns the question, the costs and the cutoff, and what triggers a recheck?	The engine, the costs and the population can each change on their own
Has anyone run this policy, or only scored it?	Only the agent case ran its policies, on tasks written for this booklet

What these data cannot show

This chapter adds no new evidence. The cases are few, each has its own label, population and assumed costs, and the pattern across them is an interpretation; another set of cases could look different. Only the agent policies were executed, on tasks written for this booklet, and no policy here was put into operation on real work. Jev stands in for one engine at one date. Consistency over time, changes between engine versions and sensitivity to signals that should not matter, which a full audit would cover, were not tested here.

Data sources

Amazon Reviews 2023Results reused from the two review chaptersWeights: sample_weight restores the star mix · License: No license stated; the booklet shows aggregates and short excerpts only
CFPB Consumer Complaint DatabaseResults reused from the complaint chapterWeights: sample_weight restores the product mix · License: Public domain; the booklet shows aggregates only
SWE-rebenchResults reused from the code-review chapterWeights: sample_weight restores the outcome mix · License: CC BY 4.0, plus the license of each source repository
Constructed patchesResults reused from the code-review chapterWeights: unweighted; evenly split by construction · License: Authored for this booklet
Constructed agent tasksResults reused from the agent chapterWeights: unweighted; the mix describes the test harness · License: Authored for this booklet

All data sources

Methods

This chapter runs nothing new. Its numbers are copied by script from the other chapters' data files.

Putting the priced decisions on one scale

The three decisions use different assumed costs, so each gap and its interval is divided by the simple alternative's dollar figure in the same case, with the sign set so that right means better for Jev. The denominators are treated as fixed, so these ranges are approximate. The percent scale helps compare direction and spread; it does not make the three assumed cost sets comparable.

How to cite

@misc{singh2026comparingandauditingdecisionsystems,
  author = {Singh, Vishal},
  title = {Comparing decision systems},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/comparing-and-auditing-decision-systems},
  note = {In Decision AI. Published 30 October 2026.}
}