Chapter 1.2

From evidence to typed questions

On 5,000 sampled coffee reviews, two decision engines and a general language model rank dissatisfied reviewers with an AUC between 0.979 and 0.983, against 0.968 for a classifier trained on other reviews, and rewording the question changes some answers more than repeating it does.

Author: Vishal Singh
Affiliation: NYU Stern
Published: 29 October 2026

Full report: every estimate with its interval, and every figure as a table.

DraftFigures and numbers may change as the analysis is rebuilt.what is still being checked

First draft. The labels are star ratings, used as a proxy; no human rated these reviews.

Bean & Basket, a coffee roaster that sells bagged coffee online, wants to know which of its reviewers are unhappy and why. The firm is illustrative; the reviews come from 2,082 coffee products sold on Amazon. This chapter poses the judgment as typed questions to two decision engines, Jev and OpenAI's Luna, and to a general language model behind the same interface, on 5,000 sampled reviews. Every review comes with the reviewer's own star rating, which serves as the check. The chapter takes no business action yet; Chapter 1.3, From probabilities to actions turns the answers into a contact policy.

On this reading task, Jev ranks unhappy reviews a little better than a word-counting classifier fit to the full labeled sample, 0.980 against 0.968 on a ranking score explained below, and well above the same classifier fit to a thousand labels (0.928). So an endpoint is a strong candidate when a firm has few labels of its own. Star ratings are only a stand-in for dissatisfaction, and the issue answers have no label to check them against.

The decision

Owner and actions: The roaster's customer-insight team decides what to ask about each review and owns the wording.
Evidence at decision time: The review's title and text.
Simplest alternative: A keyword rule, or a TF-IDF classifier trained on reviews of other coffee products.
What establishes correctness: The reviewer's star rating: one or two stars counts as dissatisfied. The issue question has no label.
When the model is wrong or cannot decide: A misread review sends the wrong signal to product and care teams; an engine error leaves the review unread.
Does the policy pay: Here, only a comparison of accuracy, cost per review and delay; prices come in the next chapter.

The data and what counts as correct

The reviews come from Amazon Reviews 2023, restricted to coffee products in the Grocery and Gourmet Food category. The sample holds the same number of reviews for each star rating, so that rare one- and two-star reviews are well represented, and each review carries a sample weight that restores the star mix of the 1,428,867 coffee reviews it was drawn from. Weighted that way, 14.2% of coffee reviews have one or two stars.

The star rating is a proxy. A reviewer can give three stars and still complain bitterly, and a one-star review can be about a late delivery. The chapter uses the rating because it exists for every review without anyone reading them, and it shows below where the proxy and the engines part ways.

Questions as asked

Each engine received the review's title and text and three typed questions: a yes-or-no question, a score on a five-level rubric and a choice among named problems.

const questions = {
  dissatisfied: { type: 'boolean',
    instructions: 'Is the reviewer dissatisfied with this coffee overall?' },
  stars: { type: 'score',
    instructions: 'Which rating level best matches this review?',
    criteria: ['Lowest level: very negative, would not buy again',
      'Mostly negative', 'Mixed or neutral', 'Mostly positive',
      'Highest level: very positive, would recommend'] },
  issue: { type: 'choice',
    instructions: 'What is the main problem the reviewer raises, if any?',
    criteria: { taste: 'Flavor, bitterness, freshness',
      roast_grind: 'Roast level or grind not as expected',
      packaging: 'Damaged, leaking or unsealed package',
      shipping: 'Late, lost or wrong item', price: 'Price or value',
      caffeine: 'Caffeine content or effect', none: 'No problem raised' } },
};

A decision engine answers with more than a label. For the yes-or-no question it returns a probability. For the choice it returns a probability for every option. For the rubric it returns a probability for every level and a score that averages the level positions by those probabilities, so a score can fall between two levels. Jev's star score falls clearly between two whole levels for 24.5% of reviews. The Haiku adapter is a general language model behind the same interface; it returns a single answer and a prompted estimate of the probability, with no distribution behind it.

Answer

Each engine returns a distribution as well as an answer

Answers to three typed questions for ten sampled coffee reviews, October 2026

Item

No taste! Hardly any taste and very weak even when I use more coffee and set my coffee maker to intense flavor. Does not taste like hazelnut at all just weak regular coffee

0.911 (index 0)

0.092 (index 1)

0.003 (index 2)

0.004 (index 3)

0.005 (index 4)

Very negative
Mostly negative
Mixed or neutral
Mostly positive
Very positive

Observed label: 1 star.

Jev 1.14.0 scored stars at 0.09 on the zero-based index scale (0 to 4), which is 1.09 stars on the rubric scale (1 to 5). The most probable level is 1 star (P = 0.91).

Show data

Data for Each engine returns a distribution as well as an answer: Review 1 (1 star), stars
Engine	Status	Answer (stars)	P(selected) or P(true)	Score, index scale	Distribution	Provider confidence statistic	Prompted estimate	Observed label
Jev 1.14.0	ok	1.09	n/a	0.09	1: 0.91; 2: 0.09; 3: 0.00; 4: 0.00; 5: 0.00	n/a	no	1
OpenAI Luna	ok	1.24	n/a	0.24	1: 0.76; 2: 0.24; 3: 0.00; 4: 0.00; 5: 0.00	n/a	no	1
Claude Haiku adapter	ok	1.50	n/a	0.50	n/a	n/a	yes	1

Pick a review and a question; the bars show how each engine spread its probability.Source: Amazon Reviews 2023, coffee products; ten reviews chosen at random, two per star rating

How the engines compare with a trained classifier

Start with the yes-or-no question. Jev, Luna, the language model and a TF-IDF classifier that counts words and word pairs and was trained on reviews of other coffee products, sort unhappy reviews from happy ones well. The useful lesson is in the small differences between them.

Figure 1

Jev, Luna and a general language model all rank dissatisfied reviews a little better than a classifier trained on other coffee reviews

AUC for dissatisfied (one or two stars), log loss and the gain in AUC over the classifier, sampled coffee reviews, weighted

Decision engine
General language model
Trained classifier

Show data

Data for Jev, Luna and a general language model all rank dissatisfied reviews a little better than a classifier trained on other coffee reviews, Ranking (AUC)
Engine	Measure	Estimate	Lower	Upper
Jev 1.14.0	auc	0.980	0.978	0.983
OpenAI Luna	auc	0.979	0.976	0.982
Claude Haiku adapter	auc	0.983	0.980	0.985
TF-IDF + logistic	auc	0.968	0.964	0.972

Read the first view for ranking: every dot sits close to the right edge, and the classifier sits a little to the left of the engines. The second view shows each engine's gain over the classifier, with intervals that stay above zero. The third view scores the probabilities themselves, and there the classifier comes out ahead. Whiskers are 95% intervals from a bootstrap that resamples products.Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

The first view scores ranking with AUC: the chance that a randomly chosen unhappy review gets a higher probability than a randomly chosen happy one. Jev reaches 0.980 and the classifier 0.968, so the engines rank better with no local fitting on these labels. Switch to the log loss view and the classifier moves ahead of Jev and Luna. Log loss also asks whether the probabilities themselves are right, and the classifier was calibrated on reviews like these. Chapter 1.3, From probabilities to actions shows what that costs once a probability sets a cutoff.

A keyword rule that flags reviews containing words such as "stale", "disappointed" or "refund" is the cheapest alternative of all. It catches 46.1% of dissatisfied reviews, and 39.4% of the reviews it flags are dissatisfied.

The Haiku adapter, a general language model, ranks reviews slightly better than Jev, by +0.0024 (95% CI 0.0012 to 0.0037). These results show no advantage for a specialized endpoint over a language model on this task.

The classifier needed labeled training reviews and a calibration step, and each prediction then costs close to nothing on a laptop. The engines needed neither, but every call costs money and time.

Engine frontier

The decision engines rank dissatisfied reviews better than a classifier, at a cost per review

Weighted AUC for dissatisfied (one or two stars) against US dollars per 1,000 reviews, October 2026

Decision engine
Classical baseline
Rule

AUC, dissatisfied (whiskers: 95% interval)

Cost per 1,000 items (USD, instruction tokens included)

* No other point has both lower cost and higher AUC, dissatisfied. The dashed line joins these points.

Highest AUC, dissatisfied: Claude Haiku adapter (decision engine): AUC, dissatisfied 0.983 (95% interval 0.980 to 0.985), $0.12 per 1,000 items, latency p50 893 ms and p95 1,285 ms from one machine, n = 5,000. Lowest cost: TF-IDF + logistic at $0 per 1,000 items. On cost and quality together, no other point beats Jev 1.14.0, Claude Haiku adapter, TF-IDF + logistic.

Show data

Data for The decision engines rank dissatisfied reviews better than a classifier, at a cost per review
Engine or method	Kind	AUC, dissatisfied	Lower 95%	Upper 95%	Cost per 1,000 items (USD)	Latency p50 (ms)	Latency p95 (ms)	Region	n	Refusals (%)	Errors (%)
Jev 1.14.0	Decision engine	0.980	0.978	0.982	$0.023	123	204	one machine	5,000	n/a	0.0
OpenAI Luna	Decision engine	0.979	0.976	0.982	$0.053	259	526	one machine	5,000	n/a	0.1
Claude Haiku adapter	Decision engine	0.983	0.980	0.985	$0.12	893	1,285	one machine	5,000	n/a	0.0
TF-IDF + logistic	Classical baseline	0.968	0.964	0.972	$0	n/a	n/a	local	5,000	n/a	n/a

Points further up rank dissatisfied reviews above satisfied ones more often; points further right cost more. Whiskers are 95% intervals from a product-level bootstrap.Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

Figure 2

Jev is the cheapest of the three models per review

Mean billed cost per thousand reviews and median latency per call, by engine, October 2026

Show data

Data for Jev is the cheapest of the three models per review, Cost
engine	measure	n	value
Jev 1.14.0	cost	5,000	0.023
OpenAI Luna	cost	5,000	0.053
Claude Haiku adapter	cost	5,000	0.124

Compare the dots: Jev costs least and the Claude Haiku adapter most. The second view shows how long a call takes. Errors were rare: 0.0% of Jev's calls failed, 0.1% of Luna's and 0.0% of the adapter's.Note: Cost is the mean billed cost at list price and latency the median of the calls made from one machine; neither carries an interval.Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

The rubric question gives a second check. Each engine's star score lands near the reviewer's own rating, off by about 0.4 stars on average for Jev, and the three models are close to one another.

Figure 3

The three models' star scores miss the reviewer's rating by similar amounts

Mean absolute error of the star score and the share of reviews whose most likely level equals the rating, sampled coffee reviews, weighted

Show data

Data for The three models' star scores miss the reviewer's rating by similar amounts, Average miss
conf_high	conf_low	engine	estimate	metric	n
0.397	0.353	Jev 1.14.0	0.374	mae	5,000
0.418	0.374	OpenAI Luna	0.395	mae	4,997
0.427	0.384	Claude Haiku adapter	0.405	mae	5,000

The dots sit close together and their whiskers overlap. The second view counts exact matches between the engine's most likely level and the rating. Whiskers are 95% intervals from a bootstrap that resamples products.Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026

Training a classifier or asking an endpoint

The choice between these two routes runs through the rest of the booklet, and neither wins everywhere. The classifier here means TF-IDF with logistic regression, one simple kind of supervised model, and these cases do not speak for every method.

	A classifier you train	A decision endpoint you ask
Defining the task	Pick a target, collect past labels, fit a model	Write the evidence, typed questions, options and rubrics
Labels you need	Enough to fit and calibrate	None to get a first answer; some to check and tune how you use it
Changing the task	A new target needs new labels and a new fit	Edit the question, then check it again, because it is now a different system
Trusting its probabilities	Depends on the labels and the population	Must be checked against your population; a returned distribution is not proof of calibration
Ongoing work	Run a pipeline and watch for drift	Maintain questions, provider access and version records, and watch for drift
Costs to count	Labeling, fitting, maintenance, serving	Integration, checking, calls, retries, maintenance
Shown in these cases	Stronger money-back prediction, product agreement level with Jev on the actual complaint mix	Better review ranking with no labels, higher contact value once recalibrated

What labels buy the classifier

The classifier above trained on 20,000 labeled reviews, which many firms do not have. To see what labels buy, the classifier was refit on smaller random subsets of those reviews and each fit was scored as a contact policy, using the costs that Chapter 1.3, From probabilities to actions derives.

Figure 4

The review classifier needs thousands of labels to approach what Jev earns with none

Net value of the contact policy per thousand reviews against labeled reviews used to train the classifier, weighted

Show data

Data for The review classifier needs thousands of labels to approach what Jev earns with none
conf_high	conf_low	labels	net
3009	0	50 labels	1772
3204	0	100 labels	2224
3758	2882	200 labels	3307
4191	3684	500 labels	4022
4559	4208	1,000 labels	4466
4673	4331	2,000 labels	4551
4877	4731	5,000 labels	4816
4986	4906	10,000 labels	4944
5058	5036	20,000 labels	5047

Read down the rows: each dot is the mean of refits on that many labeled reviews, and its whisker runs from the lowest to the highest refit, which is not a confidence interval. Each count includes the fifth held out for calibration. The vertical line is Jev, which trained on none of them. Costs are the next chapter's assumptions, and checking Jev on any review set still needs some labels.Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; ten refits per size, three at the largest; engine calls October 2026; assumed costs

Each label count covers everything the classifier used: some of those labels go to calibration and the rest to fitting, and none comes from the reviews it is scored on. Raw Jev is the reference with no local fitting, which does not mean no labels at all, because checking or recalibrating it takes some. These numbers describe this classifier, sampled and fit this way. Another product, target or class mix could move them, so treat them as a guide for a pilot and not as the labels any firm needs.

The classifier earns $3,300 with two hundred labels, $4,500 with a thousand and $4,800 with five thousand. It reaches about what Jev earns only near the full sample, at $5,000. Labels still have a job on the endpoint's side: the next chapter shows that a modest labeled sample is what fixes Jev's probabilities near the cutoff.

The wording of a question is part of the policy

A question spec is a small piece of policy, and editing it changes answers. A random subset of the sample went to Jev and Luna with one edit at a time, asked as a single question. The same subset was also sent the unedited question on its own in two separate calls, so that each edit can be compared with the change between two identical calls.

Figure 5

Removing the no-problem option changes most issue answers, while rewording or reordering changes only a few

Percent of reviews whose answer differs from the original question, after one change and after asking the same question again, Jev and Luna, weighted

Same question asked again
After the change

Show data

Data for Removing the no-problem option changes most issue answers, while rewording or reordering changes only a few, All three changes
again	again_hi	again_lo	change	changed	changed_hi	changed_lo	engine	n_again	n_changed	size
0.220	0.400	0.070	Reworded question	2.49	3.95	1.38	Jev 1.14.0	1,000	1,000	small
0	n/a	n/a	Reworded question	1.69	2.66	1.03	OpenAI Luna	999	999	small
0.390	0.670	0.150	Issue options reversed	1.86	3.01	0.930	Jev 1.14.0	1,000	1,000	small
0	n/a	n/a	Issue options reversed	2.98	3.69	2.29	OpenAI Luna	1,000	1,000	small
0.390	0.670	0.150	No-problem option removed	67.36	70.59	63.89	Jev 1.14.0	1,000	1,000	large
0	n/a	n/a	No-problem option removed	69.61	72.82	66.14	OpenAI Luna	1,000	988	large

Each row joins two shares. The open dot is the share of answers that change when the same question is simply asked again; the filled dot is the share that change after the edit. The second view zooms in on the two small edits. Whiskers are 95% intervals from a bootstrap that resamples products.Source: Amazon Reviews 2023, coffee products; 1,000 reviews drawn at random from the sample, weighted; engine calls October 2026

Rewording the question changes 2.5% of Jev's answers and reversing the options changes 1.9%, against 0.2% and 0.4% between identical calls. Removing the "no problem raised" option looks larger, at 67%, but that is almost entirely forced: Jev chose that option for 67% of these reviews, and those answers must change once it is gone. Among reviews where Jev named a problem, only 0.9% of answers change. Jev and Luna choose the same issue for 94.1% of reviews, and for 81.4% of those where at least one of them names a problem.

The three panels below let you inspect the reviews behind each edit.

Variants: question wording

Rewording the dissatisfied question changes a few of Jev's answers, more than asking again does

Share of 1,000 sampled reviews whose answer changes, weighted, October 2026

Baseline: Is the reviewer dissatisfied with this coffee overall?. Question: dissatisfied, Jev 1.14.0.

Is this reviewer unhappy with the coffee they bought? (selected)2.5% (1.4%–3.9%) 42 of 1,000 items

Axis 0% to 10% of items. Dashed rule and shaded band: repeat-run spread and its interval.

Question wording: under "Is this reviewer unhappy with the coffee they bought?", 2.5% of 1,000 items (95% interval 1.4% to 3.9%) get a different dissatisfied answer than under "Is the reviewer dissatisfied with this coffee overall?" (Jev 1.14.0), against 0.2% between 2 identical repeat calls on 1,000 items. The interval sits above the repeat-run spread.

Show data

Data for Rewording the dissatisfied question changes a few of Jev's answers, more than asking again does: change shares
Variant	Items compared	Items changed (unweighted)	Weighted change share (%)	Lower 95% (%)	Upper 95% (%)	Refusal or error
Is this reviewer unhappy with the coffee they bought?	1,000	42	2.5	1.4	3.9	0

Show data

Data for Rewording the dissatisfied question changes a few of Jev's answers, more than asking again does: answers under Is this reviewer unhappy with the coffee they bought?
Answer under Is the reviewer dissatisfied with this coffee overall?	Answer under Is this reviewer unhappy with the coffee they bought?	Items (unweighted)	Unchanged
Dissatisfied	Dissatisfied	513	yes
Not dissatisfied	Not dissatisfied	445	yes
Not dissatisfied	Dissatisfied	25	no
Dissatisfied	Not dissatisfied	17	no

Compare the change share with the share that changes between two identical calls.Source: Amazon Reviews 2023, coffee products; 1,000 reviews drawn at random from the sample; weighted

Variants: option order

Reversing the issue options changes a few of Jev's choices, more than asking again does

Share of 1,000 sampled reviews whose answer changes, weighted, October 2026

Baseline: Options in the original order. Question: issue, Jev 1.14.0.

Options in reverse order (selected)1.9% (0.9%–3.0%) 33 of 1,000 items

Axis 0% to 10% of items. Dashed rule and shaded band: repeat-run spread and its interval.

Option order: under "Options in reverse order", 1.9% of 1,000 items (95% interval 0.9% to 3.0%) get a different issue answer than under "Options in the original order" (Jev 1.14.0), against 0.4% between 2 identical repeat calls on 1,000 items. The interval sits above the repeat-run spread.

Show data

Data for Reversing the issue options changes a few of Jev's choices, more than asking again does: change shares
Variant	Items compared	Items changed (unweighted)	Weighted change share (%)	Lower 95% (%)	Upper 95% (%)	Refusal or error
Options in reverse order	1,000	33	1.9	0.9	3.0	0

Show data

Data for Reversing the issue options changes a few of Jev's choices, more than asking again does: answers under Options in reverse order
Answer under Options in the original order	Answer under Options in reverse order	Items (unweighted)	Unchanged
Taste	Taste	429	yes
No problem	No problem	314	yes
Packaging	Packaging	79	yes
Price	Price	58	yes
Roast or grind	Roast or grind	41	yes
Shipping	Shipping	26	yes
Caffeine	Caffeine	20	yes
Caffeine	Taste	9	no
Taste	No problem	4	no
Caffeine	No problem	2	no
Caffeine	Price	2	no
No problem	Roast or grind	2	no
Packaging	No problem	2	no
Price	Taste	2	no
Roast or grind	Taste	2	no
Shipping	Taste	2	no
No problem	Shipping	1	no
No problem	Taste	1	no
Packaging	Roast or grind	1	no
Price	No problem	1	no
Shipping	Roast or grind	1	no
Taste	Roast or grind	1	no

Compare the change share with the share that changes between two identical calls.Source: Amazon Reviews 2023, coffee products; 1,000 reviews drawn at random from the sample; weighted

Variants: escape option

Removing the no-problem option forces a new answer on reviews that raised no problem and leaves the others almost unchanged

Share of 1,000 sampled reviews whose answer changes, weighted, October 2026

Baseline: With a no-problem option. Question: issue, Jev 1.14.0.

Without a no-problem option (selected)67.4% (63.9%–70.6%) 325 of 1,000 items

Axis 0% to 85% of items. Dashed rule and shaded band: repeat-run spread and its interval.

Escape option: under "Without a no-problem option", 67.4% of 1,000 items (95% interval 63.9% to 70.6%) get a different issue answer than under "With a no-problem option" (Jev 1.14.0), against 0.4% between 2 identical repeat calls on 1,000 items. The interval sits above the repeat-run spread.

Show data

Data for Removing the no-problem option forces a new answer on reviews that raised no problem and leaves the others almost unchanged: change shares
Variant	Items compared	Items changed (unweighted)	Weighted change share (%)	Lower 95% (%)	Upper 95% (%)	Refusal or error
Without a no-problem option	1,000	325	67.4	63.9	70.6	0

Show data

Data for Removing the no-problem option forces a new answer on reviews that raised no problem and leaves the others almost unchanged: answers under Without a no-problem option
Answer under With a no-problem option	Answer under Without a no-problem option	Items (unweighted)	Unchanged
Taste	Taste	432	yes
No problem	Taste	236	no
Packaging	Packaging	82	yes
Price	Price	61	yes
Roast or grind	Roast or grind	41	yes
No problem	Price	40	no
Caffeine	Caffeine	32	yes
Shipping	Shipping	27	yes
No problem	Caffeine	16	no
No problem	Packaging	12	no
No problem	Shipping	8	no
No problem	Roast or grind	6	no
Caffeine	Price	1	no
Roast or grind	Caffeine	1	no
Roast or grind	Taste	1	no
Shipping	Price	1	no
Shipping	Taste	1	no
Taste	Roast or grind	1	no
Taste	Shipping	1	no

Compare the change share with the share that changes between two identical calls.Source: Amazon Reviews 2023, coffee products; 1,000 reviews drawn at random from the sample; weighted

Where the engine and the rating disagree

Jev gives a probability of dissatisfaction near one to some four- and five-star reviews, and near zero to some one-star reviews; together these confident disagreements make up 0.2% of reviews. Reading them is the fastest way to learn what the label measures. Some are the engine's mistakes. Others are reviewers who wrote a complaint and gave five stars anyway, or gave one star for a reason the question does not ask about.

Rule on a disagreement

Some confident disagreements with the star rating are the rating's mistake

Reviews where Jev's probability and the star rating point in opposite directions, October 2026

Item 1 of 8

Review 1

Flavorless. I think I might have gotten a stale batch. These barely have any scent or flavor to them like they were roasted more than a month ago.

Question: Is the reviewer dissatisfied with this coffee overall?

Is the reviewer dissatisfied with this coffee overall?DissatisfiedNot dissatisfied

Choose an option to enable the reveal.

Review 1: make your ruling first. The answers appear after you choose one of 2 options and reveal.

The data table for this item appears after your ruling.

Decide first, then reveal the engine's answer and the rating.Source: Amazon Reviews 2023, coffee products; reviews drawn at random from the confident disagreements

What these data cannot show

The label is the reviewer's star rating, which measures how the reviewer scored the purchase and not always whether they were dissatisfied with the coffee. The issue question has no label at all, so the chapter can show that engines disagree about issues but not which one is right. The reviews come from people who chose to write one, so the results describe written reviews and not all customers. Costs and latencies are from one set of calls from one machine in October 2026, and engines change under the same name.

Data sources

Amazon Reviews 20235,000 coffee reviews, a sample stratified by star rating, and reviews of other coffee products for training the classifierWeights: sample_weight restores the star mix of the coffee reviews · License: No license stated; the booklet shows aggregates and short excerpts only

All data sources

Methods

Each sampled review went to two decision engines and a general language model with the same three typed questions, and to a classifier trained on reviews of other coffee products. Answers are compared with the reviewer's own star rating.

The sample

The eval sample holds the same number of reviews for each star rating, drawn with a fixed seed from the coffee reviews. Weights restore the star mix of all coffee reviews, so a share in the text describes that population rather than the balanced sample.

Variants and repeats

A random subset of the sample was sent again with one change to the question: a reworded dissatisfied question, the issue options in reverse order, and the issue question without its no-problem option. The same subset was also sent a second time with the original question, so a change from rewording can be compared with the change from simply asking again.

Intervals

Intervals are 95% bootstrap intervals that resample products within star strata, with the same resamples for every engine, so the interval on a difference reflects the pairing.

How to cite

@misc{singh2026fromevidencetotypedquestions,
  author = {Singh, Vishal},
  title = {From evidence to typed questions},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/from-evidence-to-typed-questions},
  note = {In Decision AI. Published 29 October 2026.}
}