Part I
From judgment to action
What a decision model returns, what labels buy, and why a good ranking pays only when its probabilities can be trusted near the cutoff.
Chapters
- 1.1
Decision AI in applications and agents
An application can act on a model's judgment only through a stated policy, and in the opening bug-report example that policy routes a report to an engineer whenever the chance it can be reproduced exceeds 0.556.
DraftRecordsauthored-examples
- 1.2
From evidence to typed questions
On 5,000 sampled coffee reviews, two decision engines and a general language model rank dissatisfied reviewers with an AUC between 0.979 and 0.983, against 0.968 for a classifier trained on other reviews, and rewording the question changes some answers more than repeating it does.
DraftRecordsamazon-reviews-2023
- 1.3
From probabilities to actions
At an assumed contact cost of $6 and recovery worth $48, Jev's raw probabilities earn $5,076 per thousand reviews against $5,015 for a trained classifier, with no clear gap, and recalibrating Jev on labeled reviews raises it to $5,335, because its probabilities overstate dissatisfaction near the cutoff.
DraftRecordsamazon-reviews-2023