Data Driven Decision Making

Part III · Chapter 5

From Metrics to Counterfactuals

Every metric worth acting on hides a counterfactual you must construct, not assume.

This chapter opens Part III by reframing every metric as a question about a missing counterfactual — the outcome the same units would have shown had the action never been taken. It introduces the Decision Question Card for naming the lever, unit, horizon, comparison, and threshold before any model is fit, then builds the potential-outcomes vocabulary (ATE, ATT, and the selection-bias decomposition) and shows why randomized tests dissolve that bias while bandits trade clean measurement for lower regret. Worked cases anchor each idea: a synthetic control around Colorado's 2014 cannabis legalization, a ~1,700-store milk-pricing quasi-experiment with balance and placebo checks, and Progresso elasticity that shifts from roughly −2.23 toward −3.21 once season is omitted.

Start reading

Topics covered

Decision Question Cardpotential outcomes (ATE vs. ATT)selection-bias decompositionsynthetic controlstratified randomizationmulti-armed bandits vs. stable A/B testsbalance and placebo diagnosticsomitted-variable-bias formulaendogeneity and reverse causality

In this chapter

  1. 5.1From Metrics to DecisionsIntroduces the six-line Decision Question Card that ties a metric to a lever, unit, horizon, counterfactual, and act-or-not threshold.
  2. 5.2Causality and the CounterfactualBuilds the potential-outcomes framework, derives the ATT-plus-selection-bias split, and demonstrates a synthetic control on Colorado housing values.
  3. 5.3Experiments and A/B TestingShows why randomization erases selection bias, why intervals beat point estimates, and how milk-pricing diagnostics rescue quasi-experiments.
  4. 5.4Why Historical Data Is HardExplains the four sources of endogeneity and the omitted-variable-bias formula, visualized in Progresso soup elasticity confounded by season.