Data Driven Decision Making

Part IV · Chapter 10

Supervised Models and Business Evaluation

A model isn't evaluated until its scores meet the firm's cost matrix and ship with a card.

This chapter focuses on building, grading, and shipping the models that fill the predictive task. It opens with logistic regression as a defensible first churn scorer, then assembles the full grading toolkit — confusion matrix, ROC and PR curves, calibration, and lift — culminating in the chart a manager should read first: the threshold-profit curve that puts the firm's own cost matrix on the y-axis. From there it covers numeric prediction graded in business dollars, trees and ensembles for the interactions a linear model misses, and the AutoML-era reality that promotes the manager to task-definer and model-card author. A RentHop case ties it together, turning thousands of messy New York apartment listings into a ranked “Hot listings” queue.

Start reading

Topics covered

log-odds coefficients and odds ratiosPR-AUC vs. ROC-AUC under class imbalancecalibration and lift curvesthe threshold-profit curveMAE, RMSE, and R² for numeric errorresidual diagnostics and heteroskedasticityrandom forests and gradient boostingthe bias-variance trade-offpermutation importance, partial dependence, and SHAPmodel cards as deployment contracts

In this chapter

  1. 10.1Logistic Regression for Churn ScoringRecasts logistic regression from causal estimator to a fast, inspectable customer-ranking scorer that reads coefficients in log-odds and outputs thresholdable probabilities.
  2. 10.2Classification EvaluationAssembles the classifier grading toolkit — confusion matrix, ROC/PR, calibration, lift — and lands on the threshold-profit curve that picks the operating point.
  3. 10.3Numeric PredictionGrades predict-a-number models in business units via MAE, RMSE, and R², actual-vs-predicted plots, and residual diagnostics against the decision flow.
  4. 10.4Trees and EnsemblesIntroduces decision trees, random forests, and gradient boosting for interactions a linear model misses, anchored by the bias-variance trade-off.
  5. 10.5AutoML, Explainability, and Model CardsShows how AutoML automates the easy half while the manager defines the task, reads importance and SHAP, and writes the model card.
  6. 10.6Case Study: RentHop Hot ListingsTurns RentHop's New York listings into a Hot-apartment ranking queue via location clustering, parsed amenities, and a narrow random-forest win.