Reference
Reading guide
The rules of evidence, the recurring patterns, the parts of a figure and the units of analysis.
Kinds of claims
Each chapter says what kind of evidence stands behind a claim.
| Kind of claim | The question it asks | What a chapter can say |
|---|---|---|
| Measured performance | How often does a policy take the right action on the records in the sample, at what cost and delay? | That one policy sends more defective changes to review, or misroutes fewer complaints, than another on these records, with an interval. |
| Priced scenario | What would a policy cost a firm if the reader's cost inputs were true? | An expected loss under stated, assumed costs. It is a calculation, not an observed business return. |
| Constructed example | Does a policy handle a case written to test it? | That a policy handles or misses that case. Constructed results are labeled and never pooled with sampled ones. |
| Study with a stated identification strategy | Did a randomized experiment separate the result of an action from selection? | A causal claim, from a cited study, inside a box labeled "Causal evidence" that names the design. |
Outside a causal box a chapter uses descriptive verbs. A model that assigns a high probability of dissatisfaction to a reviewer says nothing about whether contacting that reviewer would change what the reviewer does next.
Four quantities that get confused
The probability of a label, a vendor's confidence statistic, the probability that an action succeeds and a rubric score are four different quantities. A model can be sure that a complaint is about a credit card and still lack the information needed to resolve it. The chapters keep the four apart.
How to read a figure and a scenario
A figure has a title that states its finding, a subtitle that says what is plotted and a caption that begins with what to look for. The source line names the dataset, the years and whether estimates are weighted back to the population the sample was drawn from. A whisker or band around an estimate is a 95% interval.
An interactive scenario starts from a default state whose numbers match the text. Inputs the reader sets carry the tag "assumed", and outputs computed from the data carry the tag "measured". Every scenario has a text summary and a "Show data" table, so color and position are never the only cue.
What grain means
Grain is what one row of a chapter's data describes, and it limits what the chapter can claim. Every chapter in this booklet works with records that a decision system judges: a review, a complaint, a headline, a code change or a checkpoint in an agent's work. Claims are about how policies perform on those records, not about the people who wrote them.
The methods page gives the rules every chapter shares. The data sources page lists each dataset, and the glossary defines each term of art.