# From evidence to typed questions: report
Chapter 1.2 of Decision AI, by Vishal Singh (NYU Stern). Generated from the chapter's data files; the reading version is at https://vishalsingh.org/decisionai/from-evidence-to-typed-questions. Table n of this report is figure n of the chapter.
## Identification
- Title: From evidence to typed questions
- Chapter: 1.2 (Part I, From judgment to action; Case studies)
- Question: How does the way a judgment about a review is posed change the answer a decision model returns, and how do engines compare with older classifiers?
- Dek: On 5,000 sampled coffee reviews, two decision engines and a general language model rank dissatisfied reviewers with an AUC between 0.979 and 0.983, against 0.968 for a classifier trained on other reviews, and rewording the question changes some answers more than repeating it does.
- Grain: records
- Status: draft
- Published: 2026-10-29
- Updated: not revised
- Data sources:
- Amazon Reviews 2023. 5,000 coffee reviews, a sample stratified by star rating, and reviews of other coffee products for training the classifier; weights: sample_weight restores the star mix of the coffee reviews
- Canonical URL: https://vishalsingh.org/decisionai/from-evidence-to-typed-questions
- Report: https://vishalsingh.org/decisionai/from-evidence-to-typed-questions/report
- Markdown: https://vishalsingh.org/decisionai/from-evidence-to-typed-questions/report.md
- BibTeX key: singh2026fromevidencetotypedquestions
```bibtex
@misc{singh2026fromevidencetotypedquestions,
author = {Singh, Vishal},
title = {From evidence to typed questions},
year = {2026},
url = {https://vishalsingh.org/decisionai/from-evidence-to-typed-questions},
note = {In Decision AI. Published 29 October 2026.}
}
```
## Principal estimates
The facts the dek and the opening prose cite, in order of appearance.
| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
| --- | --- | --- | ---: | --- | ---: | --- | --- | --- |
| sample size | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 5,000 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | n_reviews |
| weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 0.979 | 95% CI 0.976 to 0.982 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | auc_luna |
| weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 0.983 | 95% CI 0.980 to 0.985 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | auc_adapter |
| weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 0.968 | 95% CI 0.964 to 0.972 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | auc_tfidf |
| distinct products in the sample | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 2,082 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | n_products |
| weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 0.980 | 95% CI 0.978 to 0.982 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | auc_jev |
| training size in the small-sample refit | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | a thousand | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | lc_n_small |
| mean AUC over refits on 1,000 labeled reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 0.928 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | lc_auc_1000 |
## Findings
Bean & Basket, a coffee roaster that sells bagged coffee online, wants to know which of its reviewers are unhappy and why. The firm is illustrative; the reviews come from 2,082 coffee products sold on Amazon. This chapter poses the judgment as typed questions to two decision engines, Jev and OpenAI's Luna, and to a general language model behind the same interface, on 5,000 sampled reviews. Every review comes with the reviewer's own star rating, which serves as the check. The chapter takes no business action yet; Chapter 1.3, From probabilities to actions (https://vishalsingh.org/decisionai/from-probabilities-to-actions) turns the answers into a contact policy.
On this reading task, Jev ranks unhappy reviews a little better than a word-counting classifier fit to the full labeled sample, 0.980 against 0.968 on a ranking score explained below, and well above the same classifier fit to a thousand labels (0.928). So an endpoint is a strong candidate when a firm has few labels of its own. Star ratings are only a stand-in for dissatisfaction, and the issue answers have no label to check them against.
### The data and what counts as correct
The reviews come from Amazon Reviews 2023, restricted to coffee products in the Grocery and Gourmet Food category. The sample holds the same number of reviews for each star rating, so that rare one- and two-star reviews are well represented, and each review carries a sample weight that restores the star mix of the 1,428,867 coffee reviews it was drawn from. Weighted that way, 14.2% of coffee reviews have one or two stars.
The star rating is a proxy. A reviewer can give three stars and still complain bitterly, and a one-star review can be about a late delivery. The chapter uses the rating because it exists for every review without anyone reading them, and it shows below where the proxy and the engines part ways.
### Questions as asked
Each engine received the review's title and text and three typed questions: a yes-or-no question, a score on a five-level rubric and a choice among named problems.
```ts
const questions = {
dissatisfied: { type: 'boolean',
instructions: 'Is the reviewer dissatisfied with this coffee overall?' },
stars: { type: 'score',
instructions: 'Which rating level best matches this review?',
criteria: ['Lowest level: very negative, would not buy again',
'Mostly negative', 'Mixed or neutral', 'Mostly positive',
'Highest level: very positive, would recommend'] },
issue: { type: 'choice',
instructions: 'What is the main problem the reviewer raises, if any?',
criteria: { taste: 'Flavor, bitterness, freshness',
roast_grind: 'Roast level or grind not as expected',
packaging: 'Damaged, leaking or unsealed package',
shipping: 'Late, lost or wrong item', price: 'Price or value',
caffeine: 'Caffeine content or effect', none: 'No problem raised' } },
};
```
A decision engine answers with more than a label. For the yes-or-no question it returns a probability. For the choice it returns a probability for every option. For the rubric it returns a probability for every level and a score that averages the level positions by those probabilities, so a score can fall between two levels. Jev's star score falls clearly between two whole levels for 24.5% of reviews. The Haiku adapter is a general language model behind the same interface; it returns a single answer and a prompted estimate of the probability, with no distribution behind it.
Find a review whose score falls between two levels. What does a fractional star mean, and when would you rather have the distribution than the single most likely level?
### How the engines compare with a trained classifier
Start with the yes-or-no question. Jev, Luna, the language model and a TF-IDF classifier that counts words and word pairs and was trained on reviews of other coffee products, sort unhappy reviews from happy ones well. The useful lesson is in the small differences between them.
Table 1. Jev, Luna and a general language model all rank dissatisfied reviews a little better than a classifier trained on other coffee reviews
The first view scores ranking with AUC: the chance that a randomly chosen unhappy review gets a higher probability than a randomly chosen happy one. Jev reaches 0.980 and the classifier 0.968, so the engines rank better with no local fitting on these labels. Switch to the log loss view and the classifier moves ahead of Jev and Luna. Log loss also asks whether the probabilities themselves are right, and the classifier was calibrated on reviews like these. Chapter 1.3, From probabilities to actions (https://vishalsingh.org/decisionai/from-probabilities-to-actions) shows what that costs once a probability sets a cutoff.
A keyword rule that flags reviews containing words such as "stale", "disappointed" or "refund" is the cheapest alternative of all. It catches 46.1% of dissatisfied reviews, and 39.4% of the reviews it flags are dissatisfied.
The Haiku adapter, a general language model, ranks reviews slightly better than Jev, by +0.0024 (95% CI 0.0012 to 0.0037). These results show no advantage for a specialized endpoint over a language model on this task.
The classifier needed labeled training reviews and a calibration step, and each prediction then costs close to nothing on a laptop. The engines needed neither, but every call costs money and time.
Table 2. Jev is the cheapest of the three models per review
The rubric question gives a second check. Each engine's star score lands near the reviewer's own rating, off by about 0.4 stars on average for Jev, and the three models are close to one another.
Table 3. The three models' star scores miss the reviewer's rating by similar amounts
### Training a classifier or asking an endpoint
The choice between these two routes runs through the rest of the booklet, and neither wins everywhere. The classifier here means TF-IDF with logistic regression, one simple kind of supervised model, and these cases do not speak for every method.
| | A classifier you train | A decision endpoint you ask |
| --- | --- | --- |
| Defining the task | Pick a target, collect past labels, fit a model | Write the evidence, typed questions, options and rubrics |
| Labels you need | Enough to fit and calibrate | None to get a first answer; some to check and tune how you use it |
| Changing the task | A new target needs new labels and a new fit | Edit the question, then check it again, because it is now a different system |
| Trusting its probabilities | Depends on the labels and the population | Must be checked against your population; a returned distribution is not proof of calibration |
| Ongoing work | Run a pipeline and watch for drift | Maintain questions, provider access and version records, and watch for drift |
| Costs to count | Labeling, fitting, maintenance, serving | Integration, checking, calls, retries, maintenance |
| Shown in these cases | Stronger money-back prediction, product agreement level with Jev on the actual complaint mix | Better review ranking with no labels, higher contact value once recalibrated |
### What labels buy the classifier
The classifier above trained on 20,000 labeled reviews, which many firms do not have. To see what labels buy, the classifier was refit on smaller random subsets of those reviews and each fit was scored as a contact policy, using the costs that Chapter 1.3, From probabilities to actions (https://vishalsingh.org/decisionai/from-probabilities-to-actions) derives.
Jev's raw probabilities earn $5,076 per thousand reviews at those costs. What would you expect a classifier trained on a thousand labeled reviews to earn?
Table 4. The review classifier needs thousands of labels to approach what Jev earns with none
Each label count covers everything the classifier used: some of those labels go to calibration and the rest to fitting, and none comes from the reviews it is scored on. Raw Jev is the reference with no local fitting, which does not mean no labels at all, because checking or recalibrating it takes some. These numbers describe this classifier, sampled and fit this way. Another product, target or class mix could move them, so treat them as a guide for a pilot and not as the labels any firm needs.
The classifier earns $3,300 with two hundred labels, $4,500 with a thousand and $4,800 with five thousand. It reaches about what Jev earns only near the full sample, at $5,000. Labels still have a job on the endpoint's side: the next chapter shows that a modest labeled sample is what fixes Jev's probabilities near the cutoff.
### The wording of a question is part of the policy
A question spec is a small piece of policy, and editing it changes answers. A random subset of the sample went to Jev and Luna with one edit at a time, asked as a single question. The same subset was also sent the unedited question on its own in two separate calls, so that each edit can be compared with the change between two identical calls.
Table 5. Removing the no-problem option changes most issue answers, while rewording or reordering changes only a few
Rewording the question changes 2.5% of Jev's answers and reversing the options changes 1.9%, against 0.2% and 0.4% between identical calls. Removing the "no problem raised" option looks larger, at 67%, but that is almost entirely forced: Jev chose that option for 67% of these reviews, and those answers must change once it is gone. Among reviews where Jev named a problem, only 0.9% of answers change. Jev and Luna choose the same issue for 94.1% of reviews, and for 81.4% of those where at least one of them names a problem.
The three panels below let you inspect the reviews behind each edit.
### Where the engine and the rating disagree
Jev gives a probability of dissatisfaction near one to some four- and five-star reviews, and near zero to some one-star reviews; together these confident disagreements make up 0.2% of reviews. Reading them is the fastest way to learn what the label measures. Some are the engine's mistakes. Others are reviewers who wrote a complaint and gave five stars anyway, or gave one star for a reason the question does not ask about.
The question spec is policy. Who should own its wording, who reviews a change, and how would you version it so that last month's answers can be compared with this month's? When is a word-counting classifier still the right tool?
Ask students to predict, before showing the wording comparison, whether a synonym will change more answers than asking the same question again. Then ask why removing the no-problem option changes most answers and yet says little about how sensitive the engine is to wording.
### What these data cannot show
The label is the reviewer's star rating, which measures how the reviewer scored the purchase and not always whether they were dissatisfied with the coffee. The issue question has no label at all, so the chapter can show that engines disagree about issues but not which one is right. The reviews come from people who chose to write one, so the results describe written reviews and not all customers. Costs and latencies are from one set of calls from one machine in October 2026, and engines change under the same name.
## Tables
### Table 1. Jev, Luna and a general language model all rank dissatisfied reviews a little better than a classifier trained on other coffee reviews
AUC for dissatisfied (one or two stars), log loss and the gain in AUC over the classifier, sampled coffee reviews, weighted
| Engine | Measure | Estimate | Lower | Upper |
| --- | --- | ---: | ---: | ---: |
| Jev 1.14.0 | auc | 0.980 | 0.978 | 0.983 |
| OpenAI Luna | auc | 0.979 | 0.976 | 0.982 |
| Claude Haiku adapter | auc | 0.983 | 0.980 | 0.985 |
| TF-IDF + logistic | auc | 0.968 | 0.964 | 0.972 |
| Jev 1.14.0 | logloss | 0.179 | 0.171 | 0.187 |
| OpenAI Luna | logloss | 0.187 | 0.175 | 0.200 |
| Claude Haiku adapter | logloss | 0.165 | 0.158 | 0.173 |
| TF-IDF + logistic | logloss | 0.157 | 0.148 | 0.167 |
| Jev 1.14.0 | gain | 0.012 | 0.009 | 0.016 |
| OpenAI Luna | gain | 0.011 | 0.007 | 0.015 |
| Claude Haiku adapter | gain | 0.015 | 0.011 | 0.018 |
Views: Ranking (AUC) (metric = auc); Gain over the classifier (metric = gain); Probability fit (log loss) (metric = logloss).
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-scores.json
### Table 2. Jev is the cheapest of the three models per review
Mean billed cost per thousand reviews and median latency per call, by engine, October 2026
| engine | measure | n | value |
| --- | --- | ---: | ---: |
| Jev 1.14.0 | cost | 5,000 | 0.023 |
| Jev 1.14.0 | latency | 5,000 | 123 |
| OpenAI Luna | cost | 5,000 | 0.053 |
| OpenAI Luna | latency | 5,000 | 259 |
| Claude Haiku adapter | cost | 5,000 | 0.124 |
| Claude Haiku adapter | latency | 5,000 | 893 |
Views: Cost (measure = cost); Latency (measure = latency).
Note: Cost is the mean billed cost at list price and latency the median of the calls made from one machine; neither carries an interval.
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-cost-speed.json
### Table 3. The three models' star scores miss the reviewer's rating by similar amounts
Mean absolute error of the star score and the share of reviews whose most likely level equals the rating, sampled coffee reviews, weighted
| conf_high | conf_low | engine | estimate | metric | n |
| ---: | ---: | --- | ---: | --- | ---: |
| 0.397 | 0.353 | Jev 1.14.0 | 0.374 | mae | 5,000 |
| 0.418 | 0.374 | OpenAI Luna | 0.395 | mae | 4,997 |
| 0.427 | 0.384 | Claude Haiku adapter | 0.405 | mae | 5,000 |
| 73.98 | 70.53 | Jev 1.14.0 | 72.22 | exact | 5,000 |
| 69.78 | 65.83 | OpenAI Luna | 67.76 | exact | 4,997 |
| 74.75 | 71.54 | Claude Haiku adapter | 73.18 | exact | 5,000 |
Views: Average miss (metric = mae); Exact level (metric = exact).
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/star-score-error.json
### Table 4. The review classifier needs thousands of labels to approach what Jev earns with none
Net value of the contact policy per thousand reviews against labeled reviews used to train the classifier, weighted
| conf_high | conf_low | labels | net |
| ---: | ---: | --- | ---: |
| 3009 | 0 | 50 labels | 1772 |
| 3204 | 0 | 100 labels | 2224 |
| 3758 | 2882 | 200 labels | 3307 |
| 4191 | 3684 | 500 labels | 4022 |
| 4559 | 4208 | 1,000 labels | 4466 |
| 4673 | 4331 | 2,000 labels | 4551 |
| 4877 | 4731 | 5,000 labels | 4816 |
| 4986 | 4906 | 10,000 labels | 4944 |
| 5058 | 5036 | 20,000 labels | 5047 |
Source: Amazon Reviews 2023, coffee products; 5,000 sampled reviews, weighted; ten refits per size, three at the largest; engine calls October 2026; assumed costs
Data: https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/labels-to-catch-up.json
### Table 5. Removing the no-problem option changes most issue answers, while rewording or reordering changes only a few
Percent of reviews whose answer differs from the original question, after one change and after asking the same question again, Jev and Luna, weighted
| again | again_hi | again_lo | change | changed | changed_hi | changed_lo | engine | n_again | n_changed | size |
| ---: | ---: | ---: | --- | ---: | ---: | ---: | --- | ---: | ---: | --- |
| 0.220 | 0.400 | 0.070 | Reworded question | 2.49 | 3.95 | 1.38 | Jev 1.14.0 | 1,000 | 1,000 | small |
| 0 | n/a | n/a | Reworded question | 1.69 | 2.66 | 1.03 | OpenAI Luna | 999 | 999 | small |
| 0.390 | 0.670 | 0.150 | Issue options reversed | 1.86 | 3.01 | 0.930 | Jev 1.14.0 | 1,000 | 1,000 | small |
| 0 | n/a | n/a | Issue options reversed | 2.98 | 3.69 | 2.29 | OpenAI Luna | 1,000 | 1,000 | small |
| 0.390 | 0.670 | 0.150 | No-problem option removed | 67.36 | 70.59 | 63.89 | Jev 1.14.0 | 1,000 | 1,000 | large |
| 0 | n/a | n/a | No-problem option removed | 69.61 | 72.82 | 66.14 | OpenAI Luna | 1,000 | 988 | large |
Views: All three changes (every row); Small edits only (size = small).
Source: Amazon Reviews 2023, coffee products; 1,000 reviews drawn at random from the sample, weighted; engine calls October 2026
Data: https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/question-changes.json
## All estimates
Every fact in facts.json (37), sorted by id.
| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
| --- | ---: | --- | --- | --- | --- | --- | ---: | --- | --- | --- |
| auc_adapter | 0.983 | area under the ROC curve | weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.980 to 0.985 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| auc_diff_adapter_jev | +0.0024 | difference in AUC | general language model minus Jev | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.0012 to 0.0037 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| auc_jev | 0.980 | area under the ROC curve | weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.978 to 0.982 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| auc_luna | 0.979 | area under the ROC curve | weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.976 to 0.982 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| auc_tfidf | 0.968 | area under the ROC curve | weighted AUC for dissatisfied (one or two stars) | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.964 to 0.972 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| change_escape_jev | 67.4% | percent of reviews | answer differs between the two versions of the question | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 63.9% to 70.6% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| change_escape_named_jev | 0.9% | percent of reviews | answer differs, among reviews where Jev named a problem with the option present | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | | 682 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| change_order_jev | 1.9% | percent of reviews | answer differs between the two versions of the question | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.9% to 3.0% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| change_wording_jev | 2.5% | percent of reviews | answer differs between the two versions of the question | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 1.4% to 3.9% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| err_adapter | 0.0% | share of calls | calls that returned an error or a refusal | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| err_jev | 0.0% | share of calls | calls that returned an error or a refusal | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| err_luna | 0.1% | share of calls | calls that returned an error or a refusal | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| issue_agree | 94.1% | share of reviews | Jev and Luna choose the same issue | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| issue_agree_named | 81.4% | share of reviews | Jev and Luna choose the same issue, among reviews where at least one names a problem | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 3,412 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_auc_1000 | 0.928 | area under the ROC curve | mean AUC over refits on 1,000 labeled reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_auc_200 | 0.821 | area under the ROC curve | mean AUC over refits on 200 labeled reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_auc_20000 | 0.968 | area under the ROC curve | mean AUC over refits on 20,000 labeled reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_auc_5000 | 0.957 | area under the ROC curve | mean AUC over refits on 5,000 labeled reviews | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_n_small | a thousand | labeled reviews | training size in the small-sample refit | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_net_1000 | $4,466 | US dollars per 1,000 reviews | mean net value over refits on 1,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_net_200 | $3,307 | US dollars per 1,000 reviews | mean net value over refits on 200 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_net_20000 | $5,047 | US dollars per 1,000 reviews | mean net value over refits on 20,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_net_5000 | $4,816 | US dollars per 1,000 reviews | mean net value over refits on 5,000 labeled reviews, at assumed contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| lc_net_jev | $5,076 | US dollars per 1,000 reviews | Jev's raw probabilities at the same contact costs | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| mae_jev | 0.37 | stars | weighted mean absolute error of the score against the rating | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.35 to 0.40 | 5,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| n_confident_disagree | 0.2% | share of reviews | Jev P(dissatisfied) at least 0.9 on four or five stars, or at most 0.05 on one star | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| n_population | 1,428,867 | reviews | coffee reviews the sample stands for | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| n_products | 2,082 | products | distinct products in the sample | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| n_reviews | 5,000 | reviews | sample size | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| n_train | 20,000 | labeled reviews | classifier training sample | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| noise_order_jev | 0.4% | percent of reviews | answer differs between two identical calls | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.1% to 0.7% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| noise_wording_jev | 0.2% | percent of reviews | answer differs between two identical calls | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | 95% CI 0.1% to 0.4% | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| rule_precision | 39.4% | share of flagged reviews that are dissatisfied | weighted precision | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| rule_recall | 46.1% | share of dissatisfied reviews the rule flags | weighted recall | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| share_dissatisfied | 14.2% | share of reviews with one or two stars | weighted share | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| share_fractional_jev | 24.5% | share of reviews | Jev score at least a quarter star from a whole number | 5,000 sampled coffee reviews in Amazon Reviews 2023, Grocery and Gourmet Food | reviews written 2006 to 2023; engines called October 2026 | | | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
| share_none_jev | 67.1% | percent of reviews | Jev answers no problem raised | 1,000 reviews drawn at random from the sample | reviews written 2006 to 2023; engines called October 2026 | | 1,000 reviews | weighted to the star mix of the coffee reviews (sample_weight) | Amazon Reviews 2023 | |
## Definitions
- Sample weight: A number attached to each sampled item so that weighted results describe the population the sample was drawn from rather than the sample's own mix.
- AUC: Area under the ROC curve: the chance that a randomly chosen positive record gets a higher score than a randomly chosen negative one. It measures ranking, not calibration.
## Methods
Each sampled review went to two decision engines and a general language model with the same three typed questions, and to a classifier trained on reviews of other coffee products. Answers are compared with the reviewer's own star rating.
### The sample
The eval sample holds the same number of reviews for each star rating, drawn with a fixed seed from the coffee reviews. Weights restore the star mix of all coffee reviews, so a share in the text describes that population rather than the balanced sample.
### Variants and repeats
A random subset of the sample was sent again with one change to the question: a reworded dissatisfied question, the issue options in reverse order, and the issue question without its no-problem option. The same subset was also sent a second time with the original question, so a change from rewording can be compared with the change from simply asking again.
### Intervals
Intervals are 95% bootstrap intervals that resample products within star strata, with the same resamples for every engine, so the interval on a difference reflects the pairing.
## What these data cannot show
The label is the reviewer's star rating, which measures how the reviewer scored the purchase and not always whether they were dissatisfied with the coffee. The issue question has no label at all, so the chapter can show that engines disagree about issues but not which one is right. The reviews come from people who chose to write one, so the results describe written reviews and not all customers. Costs and latencies are from one set of calls from one machine in October 2026, and engines change under the same name.
## References
The chapter cites no reference.
## Provenance
- Build script: decision_ai/chapters/from-evidence-to-typed-questions/build.py (sha256 614bfa013c23aaad7fb46c684c28d74a47931f487d9924d97ba5996ce54c6dd0)
- Built on: 2026-10-29
- Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0
Outputs:
| File | SHA-256 |
| --- | --- |
| facts.json | f2bea860b34a80b68cba30c4778a50fb56cb894d00dbe90d0835b4239fc38f7f |
| figures/engine-cost-speed.json | f8da6d8e7a6ce6135edc3aa5e3f05cbcd27e0b423bd8aea06b90bd0395b91218 |
| figures/engine-scores.json | aa494371dfbbd60a3c5eda50ee6440fbf0b015811042f4031a8b815a2bf0d80f |
| figures/labels-to-catch-up.json | 78a2da5df8b850d5a954f667a6c17df5cff672112bec4baceda69a9ed46f8e68 |
| figures/question-changes.json | c1ce35a86b5b074236a5853a52829be7f7dafca85ef0b4fe6dfb7719ff9c7f29 |
| figures/star-score-error.json | deb07210f374f86111280492372ed21de8309d4d5f327f4a241f27afead1eecd |
| meta.json | e5508615adc538ad383df2ea75ac289022ee9b420f73a543c4e3621ae6e71773 |
| public/data/from-evidence-to-typed-questions/answer.json | 18a7a97e8c330fa254f4c34c2f6e9f0257c148dbb0f5912ef6dfd23073dfd1e7 |
| public/data/from-evidence-to-typed-questions/excerpts.json | db37452cad35c16930ad5bb119733b88e8479163f660dd69ce4f7359865cbd92 |
| public/data/from-evidence-to-typed-questions/frontier.json | 0c8b1e929247ec1535942e6a642fbdd77d65609b55fe7d06e9918568ac5f6d79 |
| public/data/from-evidence-to-typed-questions/reveal.json | 5d4a01c21a94e63e4b03b7e6e809425eec29a3733069229a090ef6f98a7a928f |
| public/data/from-evidence-to-typed-questions/variant-escape.json | d29bb4e5d3151f2ba81fc61ad61d35e9ec5d02b9b60287ee4cdadbd978452c9f |
| public/data/from-evidence-to-typed-questions/variant-order.json | 69f4f4b073d906307286fd248eb61f3ebc988ff84f7e8ceda8aa51715e43bf7e |
| public/data/from-evidence-to-typed-questions/variant-wording.json | fc107326e237d2ad30a920b80b4b19f42ce0751de5645c0ce8ff06040c283d0a |
| refs.json | ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356 |
## Machine formats
- Report (HTML): https://vishalsingh.org/decisionai/from-evidence-to-typed-questions/report
- Report (Markdown): https://vishalsingh.org/decisionai/from-evidence-to-typed-questions/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/facts.json
- Table 1, figure "engine-scores" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-scores.json
- Table 2, figure "engine-cost-speed" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/engine-cost-speed.json
- Table 3, figure "star-score-error" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/star-score-error.json
- Table 4, figure "labels-to-catch-up" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/labels-to-catch-up.json
- Table 5, figure "question-changes" (JSON): https://vishalsingh.org/decisionai/report/from-evidence-to-typed-questions/figures/question-changes.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt