From evidence to typed questions: report

Identification

Title

From evidence to typed questions

Chapter

1.2 (Part I, From judgment to action; Case studies)

Question

How does the way a judgment about a review is posed change the answer a decision model returns, and how do engines compare with older classifiers?

Dek

On 5,000 sampled coffee reviews, two decision engines and a general language model rank dissatisfied reviewers with an AUC between 0.979 and 0.983, against 0.968 for a classifier trained on other reviews, and rewording the question changes some answers more than repeating it does.

Grain

records

Status

draft

Published

29 October 2026

Updated

not revised

Data sources

Amazon Reviews 2023. 5,000 coffee reviews, a sample stratified by star rating, and reviews of other coffee products for training the classifier; weights: sample_weight restores the star mix of the coffee reviews

Canonical URL

https://vishalsingh.org/decisionai/from-evidence-to-typed-questions

Markdown

https://vishalsingh.org/decisionai/from-evidence-to-typed-questions/report.md

BibTeX

@misc{singh2026fromevidencetotypedquestions,
  author = {Singh, Vishal},
  title = {From evidence to typed questions},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/from-evidence-to-typed-questions},
  note = {In Decision AI. Published 29 October 2026.}
}

Principal estimates

The facts the dek and the opening prose cite, in order of appearance. Every fact is in the table of all estimates below.

Estimand	Population	Period	Value	Interval	n	Weights	Source	Fact id
sample size	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	5,000			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	n_reviews
weighted AUC for dissatisfied (one or two stars)	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	0.979	95% CI 0.976 to 0.982	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	auc_luna
weighted AUC for dissatisfied (one or two stars)	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	0.983	95% CI 0.980 to 0.985	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	auc_adapter
weighted AUC for dissatisfied (one or two stars)	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	0.968	95% CI 0.964 to 0.972	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	auc_tfidf
distinct products in the sample	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	2,082			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	n_products
weighted AUC for dissatisfied (one or two stars)	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	0.980	95% CI 0.978 to 0.982	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	auc_jev
training size in the small-sample refit	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	a thousand			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	lc_n_small
mean AUC over refits on 1,000 labeled reviews	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	0.928		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023	lc_auc_1000

Findings

Bean & Basket, a coffee roaster that sells bagged coffee online, wants to know which of its reviewers are unhappy and why. The firm is illustrative; the reviews come from 2,082 coffee products sold on Amazon. This chapter poses the judgment as typed questions to two decision engines, Jev and OpenAI's Luna, and to a general language model behind the same interface, on 5,000 sampled reviews. Every review comes with the reviewer's own star rating, which serves as the check. The chapter takes no business action yet; Chapter 1.3, From probabilities to actions turns the answers into a contact policy.

On this reading task, Jev ranks unhappy reviews a little better than a word-counting classifier fit to the full labeled sample, 0.980 against 0.968 on a ranking score explained below, and well above the same classifier fit to a thousand labels (0.928). So an endpoint is a strong candidate when a firm has few labels of its own. Star ratings are only a stand-in for dissatisfaction, and the issue answers have no label to check them against.

The decision

Owner and actions: The roaster's customer-insight team decides what to ask about each review and owns the wording.
Evidence at decision time: The review's title and text.
Simplest alternative: A keyword rule, or a TF-IDF classifier trained on reviews of other coffee products.
What establishes correctness: The reviewer's star rating: one or two stars counts as dissatisfied. The issue question has no label.
When the model is wrong or cannot decide: A misread review sends the wrong signal to product and care teams; an engine error leaves the review unread.
Does the policy pay: Here, only a comparison of accuracy, cost per review and delay; prices come in the next chapter.

The data and what counts as correct

The reviews come from Amazon Reviews 2023, restricted to coffee products in the Grocery and Gourmet Food category. The sample holds the same number of reviews for each star rating, so that rare one- and two-star reviews are well represented, and each review carries a sample weight that restores the star mix of the 1,428,867 coffee reviews it was drawn from. Weighted that way, 14.2% of coffee reviews have one or two stars.

The star rating is a proxy. A reviewer can give three stars and still complain bitterly, and a one-star review can be about a late delivery. The chapter uses the rating because it exists for every review without anyone reading them, and it shows below where the proxy and the engines part ways.

Questions as asked

Each engine received the review's title and text and three typed questions: a yes-or-no question, a score on a five-level rubric and a choice among named problems.

const questions = {
  dissatisfied: { type: 'boolean',
    instructions: 'Is the reviewer dissatisfied with this coffee overall?' },
  stars: { type: 'score',
    instructions: 'Which rating level best matches this review?',
    criteria: ['Lowest level: very negative, would not buy again',
      'Mostly negative', 'Mixed or neutral', 'Mostly positive',
      'Highest level: very positive, would recommend'] },
  issue: { type: 'choice',
    instructions: 'What is the main problem the reviewer raises, if any?',
    criteria: { taste: 'Flavor, bitterness, freshness',
      roast_grind: 'Roast level or grind not as expected',
      packaging: 'Damaged, leaking or unsealed package',
      shipping: 'Late, lost or wrong item', price: 'Price or value',
      caffeine: 'Caffeine content or effect', none: 'No problem raised' } },
};

A decision engine answers with more than a label. For the yes-or-no question it returns a probability. For the choice it returns a probability for every option. For the rubric it returns a probability for every level and a score that averages the level positions by those probabilities, so a score can fall between two levels. Jev's star score falls clearly between two whole levels for 24.5% of reviews. The Haiku adapter is a general language model behind the same interface; it returns a single answer and a prompted estimate of the probability, with no distribution behind it.

Answer

Each engine returns a distribution as well as an answer

Answers to three typed questions for ten sampled coffee reviews, October 2026

Item

No taste! Hardly any taste and very weak even when I use more coffee and set my coffee maker to intense flavor. Does not taste like hazelnut at all just weak regular coffee

0.911 (index 0)

0.092 (index 1)

0.003 (index 2)

0.004 (index 3)

0.005 (index 4)

Very negative
Mostly negative
Mixed or neutral
Mostly positive
Very positive

Observed label: 1 star.

Jev 1.14.0 scored stars at 0.09 on the zero-based index scale (0 to 4), which is 1.09 stars on the rubric scale (1 to 5). The most probable level is 1 star (P = 0.91).

Show data

Data for Each engine returns a distribution as well as an answer: Review 1 (1 star), stars
Engine	Status	Answer (stars)	P(selected) or P(true)	Score, index scale	Distribution	Provider confidence statistic	Prompted estimate	Observed label
Jev 1.14.0	ok	1.09	n/a	0.09	1: 0.91; 2: 0.09; 3: 0.00; 4: 0.00; 5: 0.00	n/a	no	1
OpenAI Luna	ok	1.24	n/a	0.24	1: 0.76; 2: 0.24; 3: 0.00; 4: 0.00; 5: 0.00	n/a	no	1
Claude Haiku adapter	ok	1.50	n/a	0.50	n/a	n/a	yes	1

Pick a review and a question; the bars show how each engine spread its probability.Source: Amazon Reviews 2023, coffee products; ten reviews chosen at random, two per star rating

How the engines compare with a trained classifier

Start with the yes-or-no question. Jev, Luna, the language model and a TF-IDF classifier that counts words and word pairs and was trained on reviews of other coffee products, sort unhappy reviews from happy ones well. The useful lesson is in the small differences between them.

Table 1. Jev, Luna and a general language model all rank dissatisfied reviews a little better than a classifier trained on other coffee reviews

The first view scores ranking with AUC: the chance that a randomly chosen unhappy review gets a higher probability than a randomly chosen happy one. Jev reaches 0.980 and the classifier 0.968, so the engines rank better with no local fitting on these labels. Switch to the log loss view and the classifier moves ahead of Jev and Luna. Log loss also asks whether the probabilities themselves are right, and the classifier was calibrated on reviews like these. Chapter 1.3, From probabilities to actions shows what that costs once a probability sets a cutoff.

A keyword rule that flags reviews containing words such as "stale", "disappointed" or "refund" is the cheapest alternative of all. It catches 46.1% of dissatisfied reviews, and 39.4% of the reviews it flags are dissatisfied.

The Haiku adapter, a general language model, ranks reviews slightly better than Jev, by +0.0024 (95% CI 0.0012 to 0.0037). These results show no advantage for a specialized endpoint over a language model on this task.

The classifier needed labeled training reviews and a calibration step, and each prediction then costs close to nothing on a laptop. The engines needed neither, but every call costs money and time.

Engine frontier

The decision engines rank dissatisfied reviews better than a classifier, at a cost per review

Weighted AUC for dissatisfied (one or two stars) against US dollars per 1,000 reviews, October 2026

Decision engine
Classical baseline
Rule

AUC, dissatisfied (whiskers: 95% interval)

Cost per 1,000 items (USD, instruction tokens included)

* No other point has both lower cost and higher AUC, dissatisfied. The dashed line joins these points.

Highest AUC, dissatisfied: Claude Haiku adapter (decision engine): AUC, dissatisfied 0.983 (95% interval 0.980 to 0.985), $0.12 per 1,000 items, latency p50 893 ms and p95 1,285 ms from one machine, n = 5,000. Lowest cost: TF-IDF + logistic at $0 per 1,000 items. On cost and quality together, no other point beats Jev 1.14.0, Claude Haiku adapter, TF-IDF + logistic.

Show data

Data for The decision engines rank dissatisfied reviews better than a classifier, at a cost per review
Engine or method	Kind	AUC, dissatisfied	Lower 95%	Upper 95%	Cost per 1,000 items (USD)	Latency p50 (ms)	Latency p95 (ms)	Region	n	Refusals (%)	Errors (%)
Jev 1.14.0	Decision engine	0.980	0.978	0.982	$0.023	123	204	one machine	5,000	n/a	0.0
OpenAI Luna	Decision engine	0.979	0.976	0.982	$0.053	259	526	one machine	5,000	n/a	0.1
Claude Haiku adapter	Decision engine	0.983	0.980	0.985	$0.12	893	1,285	one machine	5,000	n/a	0.0
TF-IDF + logistic	Classical baseline	0.968	0.964	0.972	$0	n/a	n/a	local	5,000	n/a	n/a

Points further up rank dissatisfied reviews above satisfied ones more often; points further right cost more. Whiskers are 95% intervals from a product-level bootstrap.Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

Table 2. Jev is the cheapest of the three models per review

The rubric question gives a second check. Each engine's star score lands near the reviewer's own rating, off by about 0.4 stars on average for Jev, and the three models are close to one another.

Table 3. The three models' star scores miss the reviewer's rating by similar amounts

Training a classifier or asking an endpoint

The choice between these two routes runs through the rest of the booklet, and neither wins everywhere. The classifier here means TF-IDF with logistic regression, one simple kind of supervised model, and these cases do not speak for every method.

	A classifier you train	A decision endpoint you ask
Defining the task	Pick a target, collect past labels, fit a model	Write the evidence, typed questions, options and rubrics
Labels you need	Enough to fit and calibrate	None to get a first answer; some to check and tune how you use it
Changing the task	A new target needs new labels and a new fit	Edit the question, then check it again, because it is now a different system
Trusting its probabilities	Depends on the labels and the population	Must be checked against your population; a returned distribution is not proof of calibration
Ongoing work	Run a pipeline and watch for drift	Maintain questions, provider access and version records, and watch for drift
Costs to count	Labeling, fitting, maintenance, serving	Integration, checking, calls, retries, maintenance
Shown in these cases	Stronger money-back prediction, product agreement level with Jev on the actual complaint mix	Better review ranking with no labels, higher contact value once recalibrated

What labels buy the classifier

The classifier above trained on 20,000 labeled reviews, which many firms do not have. To see what labels buy, the classifier was refit on smaller random subsets of those reviews and each fit was scored as a contact policy, using the costs that Chapter 1.3, From probabilities to actions derives.

Table 4. The review classifier needs thousands of labels to approach what Jev earns with none

Each label count covers everything the classifier used: some of those labels go to calibration and the rest to fitting, and none comes from the reviews it is scored on. Raw Jev is the reference with no local fitting, which does not mean no labels at all, because checking or recalibrating it takes some. These numbers describe this classifier, sampled and fit this way. Another product, target or class mix could move them, so treat them as a guide for a pilot and not as the labels any firm needs.

The classifier earns $3,300 with two hundred labels, $4,500 with a thousand and $4,800 with five thousand. It reaches about what Jev earns only near the full sample, at $5,000. Labels still have a job on the endpoint's side: the next chapter shows that a modest labeled sample is what fixes Jev's probabilities near the cutoff.

The wording of a question is part of the policy

A question spec is a small piece of policy, and editing it changes answers. A random subset of the sample went to Jev and Luna with one edit at a time, asked as a single question. The same subset was also sent the unedited question on its own in two separate calls, so that each edit can be compared with the change between two identical calls.

Table 5. Removing the no-problem option changes most issue answers, while rewording or reordering changes only a few

Rewording the question changes 2.5% of Jev's answers and reversing the options changes 1.9%, against 0.2% and 0.4% between identical calls. Removing the "no problem raised" option looks larger, at 67%, but that is almost entirely forced: Jev chose that option for 67% of these reviews, and those answers must change once it is gone. Among reviews where Jev named a problem, only 0.9% of answers change. Jev and Luna choose the same issue for 94.1% of reviews, and for 81.4% of those where at least one of them names a problem.

The three panels below let you inspect the reviews behind each edit.

Variants: question wording

Rewording the dissatisfied question changes a few of Jev's answers, more than asking again does

Share of 1,000 sampled reviews whose answer changes, weighted, October 2026

Baseline: Is the reviewer dissatisfied with this coffee overall?. Question: dissatisfied, Jev 1.14.0.

Is this reviewer unhappy with the coffee they bought? (selected)2.5% (1.4%–3.9%) 42 of 1,000 items

Axis 0% to 10% of items. Dashed rule and shaded band: repeat-run spread and its interval.

Question wording: under "Is this reviewer unhappy with the coffee they bought?", 2.5% of 1,000 items (95% interval 1.4% to 3.9%) get a different dissatisfied answer than under "Is the reviewer dissatisfied with this coffee overall?" (Jev 1.14.0), against 0.2% between 2 identical repeat calls on 1,000 items. The interval sits above the repeat-run spread.

Show data

Data for Rewording the dissatisfied question changes a few of Jev's answers, more than asking again does: change shares
Variant	Items compared	Items changed (unweighted)	Weighted change share (%)	Lower 95% (%)	Upper 95% (%)	Refusal or error
Is this reviewer unhappy with the coffee they bought?	1,000	42	2.5	1.4	3.9	0

Show data

Data for Rewording the dissatisfied question changes a few of Jev's answers, more than asking again does: answers under Is this reviewer unhappy with the coffee they bought?
Answer under Is the reviewer dissatisfied with this coffee overall?	Answer under Is this reviewer unhappy with the coffee they bought?	Items (unweighted)	Unchanged
Dissatisfied	Dissatisfied	513	yes
Not dissatisfied	Not dissatisfied	445	yes
Not dissatisfied	Dissatisfied	25	no
Dissatisfied	Not dissatisfied	17	no

Compare the change share with the share that changes between two identical calls.Source: Amazon Reviews 2023, coffee products; 1,000 reviews drawn at random from the sample; weighted

Variants: option order

Reversing the issue options changes a few of Jev's choices, more than asking again does

Share of 1,000 sampled reviews whose answer changes, weighted, October 2026

Baseline: Options in the original order. Question: issue, Jev 1.14.0.

Options in reverse order (selected)1.9% (0.9%–3.0%) 33 of 1,000 items

Axis 0% to 10% of items. Dashed rule and shaded band: repeat-run spread and its interval.

Option order: under "Options in reverse order", 1.9% of 1,000 items (95% interval 0.9% to 3.0%) get a different issue answer than under "Options in the original order" (Jev 1.14.0), against 0.4% between 2 identical repeat calls on 1,000 items. The interval sits above the repeat-run spread.

Show data

Data for Reversing the issue options changes a few of Jev's choices, more than asking again does: change shares
Variant	Items compared	Items changed (unweighted)	Weighted change share (%)	Lower 95% (%)	Upper 95% (%)	Refusal or error
Options in reverse order	1,000	33	1.9	0.9	3.0	0

Show data

Data for Reversing the issue options changes a few of Jev's choices, more than asking again does: answers under Options in reverse order
Answer under Options in the original order	Answer under Options in reverse order	Items (unweighted)	Unchanged
Taste	Taste	429	yes
No problem	No problem	314	yes
Packaging	Packaging	79	yes
Price	Price	58	yes
Roast or grind	Roast or grind	41	yes
Shipping	Shipping	26	yes
Caffeine	Caffeine	20	yes
Caffeine	Taste	9	no
Taste	No problem	4	no
Caffeine	No problem	2	no
Caffeine	Price	2	no
No problem	Roast or grind	2	no
Packaging	No problem	2	no
Price	Taste	2	no
Roast or grind	Taste	2	no
Shipping	Taste	2	no
No problem	Shipping	1	no
No problem	Taste	1	no
Packaging	Roast or grind	1	no
Price	No problem	1	no
Shipping	Roast or grind	1	no
Taste	Roast or grind	1	no

Compare the change share with the share that changes between two identical calls.Source: Amazon Reviews 2023, coffee products; 1,000 reviews drawn at random from the sample; weighted

Variants: escape option

Removing the no-problem option forces a new answer on reviews that raised no problem and leaves the others almost unchanged

Share of 1,000 sampled reviews whose answer changes, weighted, October 2026

Baseline: With a no-problem option. Question: issue, Jev 1.14.0.

Without a no-problem option (selected)67.4% (63.9%–70.6%) 325 of 1,000 items

Axis 0% to 85% of items. Dashed rule and shaded band: repeat-run spread and its interval.

Escape option: under "Without a no-problem option", 67.4% of 1,000 items (95% interval 63.9% to 70.6%) get a different issue answer than under "With a no-problem option" (Jev 1.14.0), against 0.4% between 2 identical repeat calls on 1,000 items. The interval sits above the repeat-run spread.

Show data

Data for Removing the no-problem option forces a new answer on reviews that raised no problem and leaves the others almost unchanged: change shares
Variant	Items compared	Items changed (unweighted)	Weighted change share (%)	Lower 95% (%)	Upper 95% (%)	Refusal or error
Without a no-problem option	1,000	325	67.4	63.9	70.6	0

Show data

Data for Removing the no-problem option forces a new answer on reviews that raised no problem and leaves the others almost unchanged: answers under Without a no-problem option
Answer under With a no-problem option	Answer under Without a no-problem option	Items (unweighted)	Unchanged
Taste	Taste	432	yes
No problem	Taste	236	no
Packaging	Packaging	82	yes
Price	Price	61	yes
Roast or grind	Roast or grind	41	yes
No problem	Price	40	no
Caffeine	Caffeine	32	yes
Shipping	Shipping	27	yes
No problem	Caffeine	16	no
No problem	Packaging	12	no
No problem	Shipping	8	no
No problem	Roast or grind	6	no
Caffeine	Price	1	no
Roast or grind	Caffeine	1	no
Roast or grind	Taste	1	no
Shipping	Price	1	no
Shipping	Taste	1	no
Taste	Roast or grind	1	no
Taste	Shipping	1	no

Compare the change share with the share that changes between two identical calls.Source: Amazon Reviews 2023, coffee products; 1,000 reviews drawn at random from the sample; weighted

Where the engine and the rating disagree

Jev gives a probability of dissatisfaction near one to some four- and five-star reviews, and near zero to some one-star reviews; together these confident disagreements make up 0.2% of reviews. Reading them is the fastest way to learn what the label measures. Some are the engine's mistakes. Others are reviewers who wrote a complaint and gave five stars anyway, or gave one star for a reason the question does not ask about.

Rule on a disagreement

Some confident disagreements with the star rating are the rating's mistake

Reviews where Jev's probability and the star rating point in opposite directions, October 2026

Item 1 of 8

Review 1

Flavorless. I think I might have gotten a stale batch. These barely have any scent or flavor to them like they were roasted more than a month ago.

Question: Is the reviewer dissatisfied with this coffee overall?

Is the reviewer dissatisfied with this coffee overall?DissatisfiedNot dissatisfied

Choose an option to enable the reveal.

Review 1: make your ruling first. The answers appear after you choose one of 2 options and reveal.

The data table for this item appears after your ruling.

Decide first, then reveal the engine's answer and the rating.Source: Amazon Reviews 2023, coffee products; reviews drawn at random from the confident disagreements

What these data cannot show

The label is the reviewer's star rating, which measures how the reviewer scored the purchase and not always whether they were dissatisfied with the coffee. The issue question has no label at all, so the chapter can show that engines disagree about issues but not which one is right. The reviews come from people who chose to write one, so the results describe written reviews and not all customers. Costs and latencies are from one set of calls from one machine in October 2026, and engines change under the same name.

Tables

Table 1. Jev, Luna and a general language model all rank dissatisfied reviews a little better than a classifier trained on other coffee reviews

AUC for dissatisfied (one or two stars), log loss and the gain in AUC over the classifier, sampled coffee reviews, weighted

Engine	Measure	Estimate	Lower	Upper
Jev 1.14.0	auc	0.980	0.978	0.983
OpenAI Luna	auc	0.979	0.976	0.982
Claude Haiku adapter	auc	0.983	0.980	0.985
TF-IDF + logistic	auc	0.968	0.964	0.972
Jev 1.14.0	logloss	0.179	0.171	0.187
OpenAI Luna	logloss	0.187	0.175	0.200
Claude Haiku adapter	logloss	0.165	0.158	0.173
TF-IDF + logistic	logloss	0.157	0.148	0.167
Jev 1.14.0	gain	0.012	0.009	0.016
OpenAI Luna	gain	0.011	0.007	0.015
Claude Haiku adapter	gain	0.015	0.011	0.018

Views: Ranking (AUC) (metric = auc); Gain over the classifier (metric = gain); Probability fit (log loss) (metric = logloss).

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

Data: vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-scores.json

Table 2. Jev is the cheapest of the three models per review

Mean billed cost per thousand reviews and median latency per call, by engine, October 2026

engine	measure	n	value
Jev 1.14.0	cost	5,000	0.023
Jev 1.14.0	latency	5,000	123
OpenAI Luna	cost	5,000	0.053
OpenAI Luna	latency	5,000	259
Claude Haiku adapter	cost	5,000	0.124
Claude Haiku adapter	latency	5,000	893

Views: Cost (measure = cost); Latency (measure = latency).

Note: Cost is the mean billed cost at list price and latency the median of the calls made from one machine; neither carries an interval.

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

Data: vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-cost-speed.json

Table 3. The three models' star scores miss the reviewer's rating by similar amounts

Mean absolute error of the star score and the share of reviews whose most likely level equals the rating, sampled coffee reviews, weighted

conf_high	conf_low	engine	estimate	metric	n
0.397	0.353	Jev 1.14.0	0.374	mae	5,000
0.418	0.374	OpenAI Luna	0.395	mae	4,997
0.427	0.384	Claude Haiku adapter	0.405	mae	5,000
73.98	70.53	Jev 1.14.0	72.22	exact	5,000
69.78	65.83	OpenAI Luna	67.76	exact	4,997
74.75	71.54	Claude Haiku adapter	73.18	exact	5,000

Views: Average miss (metric = mae); Exact level (metric = exact).

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

Data: vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/star-score-error.json

Table 4. The review classifier needs thousands of labels to approach what Jev earns with none

Net value of the contact policy per thousand reviews against labeled reviews used to train the classifier, weighted

conf_high	conf_low	labels	net
3009	0	50 labels	1772
3204	0	100 labels	2224
3758	2882	200 labels	3307
4191	3684	500 labels	4022
4559	4208	1,000 labels	4466
4673	4331	2,000 labels	4551
4877	4731	5,000 labels	4816
4986	4906	10,000 labels	4944
5058	5036	20,000 labels	5047

Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; ten refits per size, three at the largest; engine calls October 2026; assumed costs

Data: vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/labels-to-catch-up.json

Table 5. Removing the no-problem option changes most issue answers, while rewording or reordering changes only a few

Percent of reviews whose answer differs from the original question, after one change and after asking the same question again, Jev and Luna, weighted

again	again_hi	again_lo	change	changed	changed_hi	changed_lo	engine	n_again	n_changed	size
0.220	0.400	0.070	Reworded question	2.49	3.95	1.38	Jev 1.14.0	1,000	1,000	small
0	n/a	n/a	Reworded question	1.69	2.66	1.03	OpenAI Luna	999	999	small
0.390	0.670	0.150	Issue options reversed	1.86	3.01	0.930	Jev 1.14.0	1,000	1,000	small
0	n/a	n/a	Issue options reversed	2.98	3.69	2.29	OpenAI Luna	1,000	1,000	small
0.390	0.670	0.150	No-problem option removed	67.36	70.59	63.89	Jev 1.14.0	1,000	1,000	large
0	n/a	n/a	No-problem option removed	69.61	72.82	66.14	OpenAI Luna	1,000	988	large

Views: All three changes (every row); Small edits only (size = small).

Source: Amazon Reviews 2023, coffee products; 1,000 reviews drawn at random from the sample, weighted; engine calls October 2026

Data: vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/question-changes.json

All estimates

Every fact in facts.json (37), sorted by id. The same values as JSON: facts.json.

Fact id	Value	Unit	Estimand	Population	Period	Interval	n	Weights	Source
auc_adapter	0.983	area under the ROC curve	weighted AUC for dissatisfied (one or two stars)	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI 0.980 to 0.985	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
auc_diff_adapter_jev	+0.0024	difference in AUC	general language model minus Jev	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI 0.0012 to 0.0037	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
auc_jev	0.980	area under the ROC curve	weighted AUC for dissatisfied (one or two stars)	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI 0.978 to 0.982	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
auc_luna	0.979	area under the ROC curve	weighted AUC for dissatisfied (one or two stars)	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI 0.976 to 0.982	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
auc_tfidf	0.968	area under the ROC curve	weighted AUC for dissatisfied (one or two stars)	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI 0.964 to 0.972	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
change_escape_jev	67.4%	percent of reviews	answer differs between the two versions of the question	1,000 reviews drawn at random from the sample	reviews written 2006 to 2023; engines called October 2026	95% CI 63.9% to 70.6%	1,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
change_escape_named_jev	0.9%	percent of reviews	answer differs, among reviews where Jev named a problem with the option present	1,000 reviews drawn at random from the sample	reviews written 2006 to 2023; engines called October 2026		682 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
change_order_jev	1.9%	percent of reviews	answer differs between the two versions of the question	1,000 reviews drawn at random from the sample	reviews written 2006 to 2023; engines called October 2026	95% CI 0.9% to 3.0%	1,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
change_wording_jev	2.5%	percent of reviews	answer differs between the two versions of the question	1,000 reviews drawn at random from the sample	reviews written 2006 to 2023; engines called October 2026	95% CI 1.4% to 3.9%	1,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
err_adapter	0.0%	share of calls	calls that returned an error or a refusal	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
err_jev	0.0%	share of calls	calls that returned an error or a refusal	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
err_luna	0.1%	share of calls	calls that returned an error or a refusal	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
issue_agree	94.1%	share of reviews	Jev and Luna choose the same issue	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
issue_agree_named	81.4%	share of reviews	Jev and Luna choose the same issue, among reviews where at least one names a problem	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		3,412 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
lc_auc_1000	0.928	area under the ROC curve	mean AUC over refits on 1,000 labeled reviews	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
lc_auc_200	0.821	area under the ROC curve	mean AUC over refits on 200 labeled reviews	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
lc_auc_20000	0.968	area under the ROC curve	mean AUC over refits on 20,000 labeled reviews	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
lc_auc_5000	0.957	area under the ROC curve	mean AUC over refits on 5,000 labeled reviews	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
lc_n_small	a thousand	labeled reviews	training size in the small-sample refit	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
lc_net_1000	$4,466	US dollars per 1,000 reviews	mean net value over refits on 1,000 labeled reviews, at assumed contact costs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
lc_net_200	$3,307	US dollars per 1,000 reviews	mean net value over refits on 200 labeled reviews, at assumed contact costs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
lc_net_20000	$5,047	US dollars per 1,000 reviews	mean net value over refits on 20,000 labeled reviews, at assumed contact costs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
lc_net_5000	$4,816	US dollars per 1,000 reviews	mean net value over refits on 5,000 labeled reviews, at assumed contact costs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
lc_net_jev	$5,076	US dollars per 1,000 reviews	Jev's raw probabilities at the same contact costs	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026		5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
mae_jev	0.37	stars	weighted mean absolute error of the score against the rating	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026	95% CI 0.35 to 0.40	5,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
n_confident_disagree	0.2%	share of reviews	Jev P(dissatisfied) at least 0.9 on four or five stars, or at most 0.05 on one star	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
n_population	1,428,867	reviews	coffee reviews the sample stands for	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
n_products	2,082	products	distinct products in the sample	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
n_reviews	5,000	reviews	sample size	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
n_train	20,000	labeled reviews	classifier training sample	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
noise_order_jev	0.4%	percent of reviews	answer differs between two identical calls	1,000 reviews drawn at random from the sample	reviews written 2006 to 2023; engines called October 2026	95% CI 0.1% to 0.7%	1,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
noise_wording_jev	0.2%	percent of reviews	answer differs between two identical calls	1,000 reviews drawn at random from the sample	reviews written 2006 to 2023; engines called October 2026	95% CI 0.1% to 0.4%	1,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
rule_precision	39.4%	share of flagged reviews that are dissatisfied	weighted precision	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
rule_recall	46.1%	share of dissatisfied reviews the rule flags	weighted recall	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
share_dissatisfied	14.2%	share of reviews with one or two stars	weighted share	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
share_fractional_jev	24.5%	share of reviews	Jev score at least a quarter star from a whole number	5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food	reviews written 2006 to 2023; engines called October 2026			weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023
share_none_jev	67.1%	percent of reviews	Jev answers no problem raised	1,000 reviews drawn at random from the sample	reviews written 2006 to 2023; engines called October 2026		1,000 reviews	weighted to the star mix of the coffee reviews (sample_weight)	Amazon Reviews 2023

Definitions

Sample weight: A number attached to each sampled item so that weighted results describe the population the sample was drawn from rather than the sample's own mix.
AUC: Area under the ROC curve: the chance that a randomly chosen positive record gets a higher score than a randomly chosen negative one. It measures ranking, not calibration.

Methods

Each sampled review went to two decision engines and a general language model with the same three typed questions, and to a classifier trained on reviews of other coffee products. Answers are compared with the reviewer's own star rating.

The sample

The eval sample holds the same number of reviews for each star rating, drawn with a fixed seed from the coffee reviews. Weights restore the star mix of all coffee reviews, so a share in the text describes that population rather than the balanced sample.

Variants and repeats

A random subset of the sample was sent again with one change to the question: a reworded dissatisfied question, the issue options in reverse order, and the issue question without its no-problem option. The same subset was also sent a second time with the original question, so a change from rewording can be compared with the change from simply asking again.

Intervals

Intervals are 95% bootstrap intervals that resample products within star strata, with the same resamples for every engine, so the interval on a difference reflects the pairing.

What these data cannot show

The label is the reviewer's star rating, which measures how the reviewer scored the purchase and not always whether they were dissatisfied with the coffee. The issue question has no label at all, so the chapter can show that engines disagree about issues but not which one is right. The reviews come from people who chose to write one, so the results describe written reviews and not all customers. Costs and latencies are from one set of calls from one machine in October 2026, and engines change under the same name.

References

The chapter cites no reference.

Provenance

Build script: decision_ai/chapters/from-evidence-to-typed-questions/build.py (sha256 614bfa013c23aaad7fb46c684c28d74a47931f487d9924d97ba5996ce54c6dd0)
Built on: 2026-10-29
Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0

Outputs

File	SHA-256
facts.json	f2bea860b34a80b68cba30c4778a50fb56cb894d00dbe90d0835b4239fc38f7f
figures/engine-cost-speed.json	f8da6d8e7a6ce6135edc3aa5e3f05cbcd27e0b423bd8aea06b90bd0395b91218
figures/engine-scores.json	aa494371dfbbd60a3c5eda50ee6440fbf0b015811042f4031a8b815a2bf0d80f
figures/labels-to-catch-up.json	78a2da5df8b850d5a954f667a6c17df5cff672112bec4baceda69a9ed46f8e68
figures/question-changes.json	c1ce35a86b5b074236a5853a52829be7f7dafca85ef0b4fe6dfb7719ff9c7f29
figures/star-score-error.json	deb07210f374f86111280492372ed21de8309d4d5f327f4a241f27afead1eecd
meta.json	e5508615adc538ad383df2ea75ac289022ee9b420f73a543c4e3621ae6e71773
public/data/from-evidence-to-typed-questions/answer.json	18a7a97e8c330fa254f4c34c2f6e9f0257c148dbb0f5912ef6dfd23073dfd1e7
public/data/from-evidence-to-typed-questions/excerpts.json	db37452cad35c16930ad5bb119733b88e8479163f660dd69ce4f7359865cbd92
public/data/from-evidence-to-typed-questions/frontier.json	0c8b1e929247ec1535942e6a642fbdd77d65609b55fe7d06e9918568ac5f6d79
public/data/from-evidence-to-typed-questions/reveal.json	5d4a01c21a94e63e4b03b7e6e809425eec29a3733069229a090ef6f98a7a928f
public/data/from-evidence-to-typed-questions/variant-escape.json	d29bb4e5d3151f2ba81fc61ad61d35e9ec5d02b9b60287ee4cdadbd978452c9f
public/data/from-evidence-to-typed-questions/variant-order.json	69f4f4b073d906307286fd248eb61f3ebc988ff84f7e8ceda8aa51715e43bf7e
public/data/from-evidence-to-typed-questions/variant-wording.json	fc107326e237d2ad30a920b80b4b19f42ce0751de5645c0ce8ff06040c283d0a
refs.json	ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356

Machine formats

Report (Markdown): https://vishalsingh.org/decisionai/from-evidence-to-typed-questions/report.md
Every estimate (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/facts.json
Table 1, figure "engine-scores" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-scores.json
Table 2, figure "engine-cost-speed" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-cost-speed.json
Table 3, figure "star-score-error" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/star-score-error.json
Table 4, figure "labels-to-catch-up" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/labels-to-catch-up.json
Table 5, figure "question-changes" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/question-changes.json
Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt