{
  "alt": "Lollipop chart of the share of money-back complaints caught by fast-tracking each model's top tenth. The trained classifier has the longest stem and the other three are close to one another.",
  "caption": "Each stem shows how many money-back complaints a desk would catch if it fast-tracked the tenth of complaints the model ranks highest. Picking a tenth at random would catch about a tenth of them.",
  "data": [
    {
      "caught": 58.2,
      "kind": "Decision engine",
      "model": "Jev 1.14.0"
    },
    {
      "caught": 63.1,
      "kind": "Decision engine",
      "model": "OpenAI Luna"
    },
    {
      "caught": 51.8,
      "kind": "General language model",
      "model": "Claude Haiku adapter"
    },
    {
      "caught": 73.1,
      "kind": "Trained classifier",
      "model": "TF-IDF + logistic"
    }
  ],
  "id": "money-back-caught",
  "notes": [
    "Ranked over all complaints, the classifier's AUC is {fact:auc_money_tfidf} and Jev's is {fact:auc_money_jev}; Jev minus the classifier is {fact:auc_money_diff_jev_tfidf}."
  ],
  "source": "CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026",
  "spec": {
    "colors": {
      "Decision engine": "cat1",
      "General language model": "cat3",
      "Trained classifier": "cat2"
    },
    "group": "kind",
    "reference": 10,
    "referenceLabel": "A random tenth",
    "sort": "desc",
    "stem": true,
    "x": "caught",
    "xDomain": [
      0,
      100
    ],
    "xFormat": {
      "digits": 1
    },
    "xLabel": "Money-back complaints caught (%)",
    "y": "model"
  },
  "subtitle": "Share of complaints closed with monetary relief that fall in each model's top tenth by P(money back), percent",
  "title": "A classifier trained on past outcomes finds money-back complaints better than the models that only read the complaint",
  "type": "dot"
}