Chapter 2.1 · Report

Resolving and routing customer complaints

A classifier trained on earlier complaints matches consumers' product choices about as often as Jev once it is fit to the actual mix, and it finds more of the complaints that end in money back, where it passes Jev with far fewer labels than the review task needs.

Chapter page: the same findings with every figure drawn as a chart.

Author: Vishal Singh
Affiliation: NYU Stern
Published: 29 October 2026

Every estimate of the chapter with its interval and every figure as a table, generated from the chapter's data files. Table n of this report is figure n of the chapter. The same report as Markdown: report.md.

Identification

Title

Resolving and routing customer complaints

Chapter

2.1 (Part II, Scarce attention; Case studies)

Question

Which product team should receive each complaint, when should a person check it first, and can a model tell which complaints will end in money back?

Dek

Grain

records

Status

draft

Published

29 October 2026

Updated

not revised

Data sources

CFPB Consumer Complaint Database. 10,000 complaints received in 2025 with a narrative of at most 3,000 characters, stratified by product; complaints from August 2023 to 2024 for the classifier; weights: sample_weight restores the product mix of the 2025 complaints within the length limit

Canonical URL

https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints

Markdown

https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints/report.md

BibTeX

@misc{singh2026resolvingandroutingcustomercomplaints,
  author = {Singh, Vishal},
  title = {Resolving and routing customer complaints},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints},
  note = {In Decision AI. Published 29 October 2026.}
}

Principal estimates

The facts the dek and the opening prose cite, in order of appearance. Every fact is in the table of all estimates below.

Estimand	Population	Period	Value	Interval	n	Weights	Source	Fact id
sample size	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	10,000			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	n_complaints
2025 complaints within the length limit that the sample stands for	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	519,841			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	n_population
top product equals the consumer-selected product	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	81.9%	95% CI 80.5% to 83.2%	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	agree_jev
top product equals the consumer-selected product	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	82.4%	95% CI 81.2% to 83.7%	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	agree_tfidf

Findings

A bank's complaint desk gets a steady stream of written complaints, and each one needs to reach the right team. Send a mortgage complaint to the credit card team and it sits for days before someone sends it back. This chapter asks a decision engine to read each complaint and pick the product it is about, then asks the question every desk manager asks next: when should a person check the engine's pick before it goes out?

The complaints are 10,000 sampled from distinct narratives filed with the Consumer Financial Protection Bureau in 2025, standing for 519,841 narratives; repeated narratives were removed, which matters because many credit-reporting complaints use the same text. No one rated them by hand. Instead, each complaint carries two answers that already exist: the product the consumer selected when filing, and what the company eventually did about it.

Jev and a classifier fit to the actual complaint mix match the consumer's product choice about equally often, 82% and 82%, and once both are recalibrated the cost comparison shows no clear winner at the assumed costs. So the choice between them rests on what a desk can run and check, and either one still needs people for uncertain complaints. Consumer choices are sometimes wrong, and the costs and review times are assumptions. The chapter follows the desk from automatic routing to human review, a limit on review time and how to spend it, then takes up one companion question: which complaints end in money back.

The decision

Owner and actions: The complaint desk lead decides where each complaint goes; operations owns the cost of delay.
Evidence at decision time: The complaint narrative as the consumer wrote it, and nothing added later.
Simplest alternative: A word-counting classifier trained on complaints from earlier years.
What establishes correctness: The consumer's own product choice, which is sometimes wrong, and the company's recorded response.
When the model is wrong or cannot decide: A misrouted complaint waits days for the right team; a person reviewing every complaint costs staff time.
Does the policy pay: Expected cost per thousand complaints under stated costs for misroutes and triage.

Picking the product

Each engine saw the narrative and a list of eleven products with a short definition of each, and picked one. Jev and Luna also returned a probability for every product; the Haiku adapter returned only its pick. The chart compares how often the top pick matches what the consumer chose.

Table 1. A classifier trained on earlier complaints agrees with consumers' product choices about as often as Jev

Jev agrees on 82% of complaints and the classifier on 82%, a gap of −0.6 (95% CI −1.9 to 0.7) points. How the classifier is trained decides this comparison. The training sample holds equal numbers of each product, and fitting it without weights drops the classifier to 78%, while the weighted fit does poorly on rare products: counted equally across products, Jev scores 75% and the classifier 64%. Jev costs about $0.05 per thousand complaints.

Agreement is a soft score here. Consumers choose their own product from a form, and some choose badly: a complaint about a debt collector chasing a credit card balance can honestly belong to either team. The flows below show where Jev and the consumers part ways. The largest disagreements run between credit reporting and debt collection, two products that often appear in the same complaint.

Routing flows

Most complaints land in the product the consumer chose; the largest other flows run between credit reporting and debt collection

Weighted flows from consumer-selected product to Jev's predicted product, 2025 complaints

Same category (solid)
Another category (hatched)

Select a category (click, or Tab to it and press Enter) to highlight its flows; Escape or a second press clears.

81.9% of 9,998 items (weighted) are predicted in their recorded category by Jev 1.14.0. The largest flow to another category is Debt collection to Credit reporting, 2.8% of all items.

Show data

Data for Most complaints land in the product the consumer chose; the largest other flows run between credit reporting and debt collection
Consumer-selected product	Jev's predicted product	Same category	Share of all items, weighted (%)	Share of the consumer-selected product row, weighted (%)	Items (unweighted)
Checking or savings	Checking or savings	yes	6.9	80.7	734
Checking or savings	Credit card	no	0.6	6.5	59
Checking or savings	Credit reporting	no	0.1	1.0	9
Checking or savings	Debt collection	no	0.0	0.6	5
Checking or savings	Money transfer	no	0.7	8.8	80
Checking or savings	Mortgage	no	0.0	0.1	1
Checking or savings	Payday or personal loan	no	0.1	0.7	6
Checking or savings	Prepaid card	no	0.1	1.3	12
Checking or savings	Vehicle loan or lease	no	0.0	0.3	3
Credit card	Checking or savings	no	0.2	2.9	26
Credit card	Credit card	yes	5.9	84.4	766
Credit card	Credit reporting	no	0.5	6.5	59
Credit card	Debt collection	no	0.2	2.4	22
Credit card	Debt or credit management	no	0.0	0.2	2
Credit card	Money transfer	no	0.1	1.4	13
Credit card	Mortgage	no	0.0	0.1	1
Credit card	Payday or personal loan	no	0.1	1.1	10
Credit card	Prepaid card	no	0.1	1.0	9
Credit reporting	Checking or savings	no	0.4	0.7	6
Credit reporting	Credit card	no	1.7	3.1	28
Credit reporting	Credit reporting	yes	46.6	84.5	769
Credit reporting	Debt collection	no	2.7	4.8	44
Credit reporting	Debt or credit management	no	0.1	0.1	1
Credit reporting	Money transfer	no	0.1	0.2	2
Credit reporting	Mortgage	no	0.4	0.7	6
Credit reporting	Payday or personal loan	no	0.5	1.0	9
Credit reporting	Student loan	no	1.4	2.5	23
Credit reporting	Vehicle loan or lease	no	1.3	2.4	22
Debt collection	Checking or savings	no	0.0	0.2	2
Debt collection	Credit card	no	0.7	5.1	46
Debt collection	Credit reporting	no	2.8	20.6	187
Debt collection	Debt collection	yes	9.5	70.3	639
Debt collection	Debt or credit management	no	0.0	0.3	3
Debt collection	Money transfer	no	0.0	0.2	2
Debt collection	Mortgage	no	0.0	0.3	3
Debt collection	Payday or personal loan	no	0.2	1.2	11
Debt collection	Student loan	no	0.1	0.4	4
Debt collection	Vehicle loan or lease	no	0.2	1.3	12
Debt or credit management	Checking or savings	no	0.0	3.6	33
Debt or credit management	Credit card	no	0.0	8.5	78
Debt or credit management	Credit reporting	no	0.1	20.9	190
Debt or credit management	Debt collection	no	0.1	21.1	191
Debt or credit management	Debt or credit management	yes	0.2	33.0	300
Debt or credit management	Money transfer	no	0.0	2.1	20
Debt or credit management	Mortgage	no	0.0	2.8	26
Debt or credit management	Payday or personal loan	no	0.0	3.8	35
Debt or credit management	Prepaid card	no	0.0	0.6	5
Debt or credit management	Student loan	no	0.0	1.5	13
Debt or credit management	Vehicle loan or lease	no	0.0	1.9	18
Money transfer	Checking or savings	no	0.7	9.9	90
Money transfer	Credit card	no	0.2	2.8	25
Money transfer	Credit reporting	no	0.0	0.2	2
Money transfer	Debt collection	no	0.0	0.1	1
Money transfer	Debt or credit management	no	0.0	0.1	1
Money transfer	Money transfer	yes	6.0	85.6	777
Money transfer	Payday or personal loan	no	0.0	0.5	5
Money transfer	Prepaid card	no	0.0	0.4	4
Money transfer	Vehicle loan or lease	no	0.0	0.3	3
Mortgage	Checking or savings	no	0.0	0.9	8
Mortgage	Credit card	no	0.0	0.6	6
Mortgage	Credit reporting	no	0.0	1.0	9
Mortgage	Debt collection	no	0.0	1.4	13
Mortgage	Debt or credit management	no	0.0	0.1	1
Mortgage	Money transfer	no	0.0	0.2	2
Mortgage	Mortgage	yes	2.2	93.0	845
Mortgage	Payday or personal loan	no	0.1	2.2	20
Mortgage	Student loan	no	0.0	0.1	1
Mortgage	Vehicle loan or lease	no	0.0	0.4	4
Payday or personal loan	Checking or savings	no	0.0	2.5	23
Payday or personal loan	Credit card	no	0.1	5.3	48
Payday or personal loan	Credit reporting	no	0.1	5.9	54
Payday or personal loan	Debt collection	no	0.1	9.7	88
Payday or personal loan	Debt or credit management	no	0.0	0.6	6
Payday or personal loan	Money transfer	no	0.0	0.6	6
Payday or personal loan	Mortgage	no	0.0	1.8	17
Payday or personal loan	Payday or personal loan	yes	1.0	68.6	623
Payday or personal loan	Prepaid card	no	0.0	0.1	1
Payday or personal loan	Student loan	no	0.0	0.8	7
Payday or personal loan	Vehicle loan or lease	no	0.1	4.0	36
Prepaid card	Checking or savings	no	0.1	12.0	109
Prepaid card	Credit card	no	0.1	10.6	96
Prepaid card	Credit reporting	no	0.0	0.3	3
Prepaid card	Debt collection	no	0.0	0.7	6
Prepaid card	Money transfer	no	0.1	16.8	153
Prepaid card	Mortgage	no	0.0	0.3	2
Prepaid card	Payday or personal loan	no	0.0	0.7	7
Prepaid card	Prepaid card	yes	0.4	58.6	533
Student loan	Checking or savings	no	0.0	0.1	1
Student loan	Credit card	no	0.0	0.9	8
Student loan	Credit reporting	no	0.1	6.4	58
Student loan	Debt collection	no	0.1	2.8	25
Student loan	Money transfer	no	0.0	0.1	1
Student loan	Mortgage	no	0.0	0.2	2
Student loan	Payday or personal loan	no	0.0	2.4	22
Student loan	Prepaid card	no	0.0	0.2	2
Student loan	Student loan	yes	1.7	86.9	790
Vehicle loan or lease	Checking or savings	no	0.0	0.5	5
Vehicle loan or lease	Credit card	no	0.1	2.6	24
Vehicle loan or lease	Credit reporting	no	0.2	7.6	69
Vehicle loan or lease	Debt collection	no	0.1	3.5	32
Vehicle loan or lease	Debt or credit management	no	0.0	0.1	1
Vehicle loan or lease	Money transfer	no	0.0	0.1	1
Vehicle loan or lease	Mortgage	no	0.0	0.4	4
Vehicle loan or lease	Payday or personal loan	no	0.1	3.4	31
Vehicle loan or lease	Prepaid card	no	0.0	0.1	1
Vehicle loan or lease	Student loan	no	0.0	0.2	2
Vehicle loan or lease	Vehicle loan or lease	yes	1.7	81.3	739

Follow the bands that change product. Some are engine errors and some are consumers who picked the wrong product; these data cannot tell which.Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted

When to ask a person first

Suppose a misrouted complaint costs $30 in delay and sending a complaint to a person for triage costs $6 of staff time. Routing pays when the chance the pick is right is high enough that the expected misroute cost falls below the triage cost. At these costs the cutoff is 0.8: route when the model is more sure than that, and send everything else to a person.

Triage band

Automate, review or triage

Loading data

The next chart shows what each policy does with the whole stream of complaints.

Table 2. At the same cutoff, Jev sends fewer complaints to people and misroutes more, because its stated probabilities run high

With the cutoff taken at face value, Jev sends 16% of complaints to a person and misroutes 10%, and the classifier sends 43% and misroutes 4%. Priced at the assumed costs, that is $4,054 per thousand complaints for Jev and $3,685 for the classifier. Routing everything by Jev with no check costs $5,442, and sending everything to a person costs $6,000.

The gap of +$368 (95% CI $97 to $655) is about calibration more than about reading. Jev's stated probabilities run high, so it routes complaints it should send to a person. Recalibrate both models on other folds of the sample and Jev costs $3,471, sending 35% to a person, against $3,666 for the classifier.

Policy comparison

With a cutoff, every model sends a share of complaints to people and misroutes fewer

Expected cost per 1,000 complaints, misroute and triage shares by routing policy, with 95% intervals; 2025 complaints, weighted

Policy	derived	measured	measured
Jev, triage below the cutoff (chapter policy)	$4,054 ($3,740–$4,384)	10.2% (9.2%–11.3%)	16.4% (15.1%–17.8%)
Luna, triage below the cutoff	$4,127 ($3,797–$4,457)	12.6% (11.5%–13.7%)	5.9% (5.1%–6.7%)
TF-IDF classifier, triage below the cutoff	$3,685 ($3,482–$3,893)	3.8% (3.1%–4.4%)	42.5% (40.8%–44.2%)
Jev recalibrated, triage below the cutoff	$3,471 ($3,243–$3,709)	4.6% (3.8%–5.3%)	35.1% (33.4%–36.9%)
TF-IDF recalibrated, triage below the cutoff	$3,666 ($3,466–$3,885)	4.0% (3.4%–4.7%)	41.0% (39.4%–42.7%)
Jev, route everything	$5,442 ($5,041–$5,861)	18.1% (16.8%–19.5%)	0.0% (0.0%–0.0%)
Haiku adapter, route everything (no probabilities)	$4,683 ($4,354–$5,011)	15.6% (14.5%–16.7%)	0.0% (0.0%–0.0%)
Send everything to triage	$6,000 ($6,000–$6,000)	0.0% (0.0%–0.0%)	100.0% (100.0%–100.0%)

n = 10,000 items. Intervals are paired bootstrap 95% intervals from the pipeline.

Jev, triage below the cutoff: expected cost per 1,000 $4,054 (95% interval $3,740 to $4,384). Jev recalibrated, triage below the cutoff has the lowest point estimate on this measure, $3,471 (95% interval $3,243 to $3,709).

Show data

Data for With a cutoff, every model sends a share of complaints to people and misroutes fewer
Policy	Expected cost per 1,000	Lower 95%	Upper 95%	Misrouted	Lower 95%	Upper 95%	Sent to triage	Lower 95%	Upper 95%
Jev, triage below the cutoff	4054	3740	4384	10.2	9.2	11.3	16.4	15.1	17.8
Luna, triage below the cutoff	4127	3797	4457	12.6	11.5	13.7	5.9	5.1	6.7
TF-IDF classifier, triage below the cutoff	3685	3482	3893	3.8	3.1	4.4	42.5	40.8	44.2
Jev recalibrated, triage below the cutoff	3471	3243	3709	4.6	3.8	5.3	35.1	33.4	36.9
TF-IDF recalibrated, triage below the cutoff	3666	3466	3885	4.0	3.4	4.7	41.0	39.4	42.7
Jev, route everything	5442	5041	5861	18.1	16.8	19.5	0.0	0.0	0.0
Haiku adapter, route everything (no probabilities)	4683	4354	5011	15.6	14.5	16.7	0.0	0.0	0.0
Send everything to triage	6000	6000	6000	0.0	0.0	0.0	100.0	100.0	100.0

Read across a row: what a policy costs and where its cost comes from. Costs are assumed; misroutes are counted against the consumer's own product choice.Source: CFPB Consumer Complaint Database, 2025 complaints; 10,000 sampled, weighted; engine calls October 2026

That comparison gives both models plenty of labels. A desk starting out would have few, so the next chart gives both the same labeled complaints from earlier years.

Table 3. For the same labels, recalibrating Jev costs less than fitting a classifier at every size tried

With five hundred labels, recalibrated Jev costs $3,600 per thousand complaints, against $4,100 for raw Jev and $4,800 for a classifier fit on those labels. With the full sample the classifier comes closer, at $3,700 against $3,500. Labels from earlier years are what a desk would really have, and these figures leave out what each model costs to run.

When reviewer time is fixed

So far a person has been free to read as many complaints as the cutoff sends. Suppose the desk instead has 400 reviewer minutes for every thousand complaints, and everything outside the queue goes where Jev's pick sends it. Which complaints should a person read? The complaints and the costs stay as they were; what changes is that the desk can no longer send a person everything below the cutoff.

Uncertainty is one reason to look at a complaint. Consequence is another, so the queue below also uses a second model. Its estimate of money back, from the companion section that follows, stands in for a misroute that matters more, under an assumed extra cost. It is a proxy and not a measured loss from a misroute. Read the columns separately, because review workload, misroutes, misrouted money-back complaints and assumed dollars can move in different directions.

Policy comparison

With reviewer time fixed, ranking by consequence and length changes which misroutes get caught

Outcomes per 1,000 complaints with 400 reviewer minutes per 1,000 complaints spent on Jev's routing picks, with 95% intervals; 2025 complaints, weighted

Policy	derived	measured	measured	derived
Review by expected loss per minute (chapter policy)	20.2% (18.8%–21.6%)	98.2 (87.0–109.5)	1.06 (0.50–1.73)	$3,074 ($2,732–$3,429)
No review	0.0% (0.0%–0.0%)	181.5 (168.2–195.6)	3.18 (2.26–4.20)	$5,827 ($5,402–$6,277)
Review the least certain picks	14.3% (13.1%–15.6%)	110.4 (99.4–121.6)	1.57 (0.89–2.31)	$3,499 ($3,144–$3,852)
Review by expected loss	14.1% (13.0%–15.3%)	114.7 (103.6–126.1)	1.14 (0.54–1.88)	$3,579 ($3,229–$3,944)

n = 10,000 items. Intervals are paired bootstrap 95% intervals from the pipeline.

Review by expected loss per minute: complaints reviewed 20.2% (95% interval 18.8% to 21.6%). It has the highest point estimate on this measure.

Show data

Data for With reviewer time fixed, ranking by consequence and length changes which misroutes get caught
Policy	Complaints reviewed	Lower 95%	Upper 95%	Misroutes left	Lower 95%	Upper 95%	Misrouted money-back complaints left	Lower 95%	Upper 95%	Expected cost	Lower 95%	Upper 95%
Review by expected loss per minute	20.2	18.8	21.6	98.2	87.0	109.5	1.06	0.50	1.73	3074	2732	3429
No review	0.0	0.0	0.0	181.5	168.2	195.6	3.18	2.26	4.20	5827	5402	6277
Review the least certain picks	14.3	13.1	15.6	110.4	99.4	121.6	1.57	0.89	2.31	3499	3144	3852
Review by expected loss	14.1	13.0	15.3	114.7	103.6	126.1	1.14	0.54	1.88	3579	3229	3944

Read down each column. Every queue spends the same reviewer minutes, and complaints outside it go where Jev's pick sends them. Review time, reviewer accuracy and costs are assumed; misroutes are counted against the consumer's product.Source: CFPB Consumer Complaint Database, 2025 distinct narratives; 10,000 sampled, weighted; engine calls October 2026; assumed review time and costs

Reviewing the least certain picks is the natural rule, and it leaves 110.4 misroutes per thousand complaints, down from 181.5 with no review. Adding the consequence moves attention toward complaints that may involve money, leaving 1.14 misrouted money-back complaints against 1.57, a difference of −0.43 (95% CI −0.91 to 0.09), whose range includes zero. Adding review time changes more: because short complaints take less time, the same minutes cover 20% of complaints instead of 14%, and the expected cost changes by −425 (95% CI −639 to −198) dollars per thousand. That gain rests on the assumed link between length and review time, and on a short complaint being reviewed as accurately as a long one. A desk would need to measure both before running this queue.

A companion question: which complaints end in money back

This is a different job from routing: it predicts what a company will do later, not what a complaint is about. A desk might fast-track complaints that are likely to end with the company paying the consumer back. Only 3% of these complaints closed with monetary relief, so finding them early matters. Each engine was asked whether the consumer describes money the company could return, and the classifier was trained directly on which earlier complaints ended in relief.

Table 4. A classifier trained on past outcomes finds money-back complaints better than the models that only read the complaint

Fast-tracking the tenth of complaints the classifier ranks highest would catch 73% of the money-back cases, against 58% for Jev, a gap of +14.9 (95% CI 8.9 to 20.4) points. Much of any model's score here comes from knowing the product, because money back is far more common for some products than others: ranking complaints by their product's past rate alone gives an AUC of 0.906, above Jev's 0.895. Within a product, the classifier's lead narrows to 0.769 against 0.742, a difference of +0.027 (95% CI −0.009 to 0.065). The engine answers the question it was asked, whether the consumer describes a refundable loss. The classifier learned from history which complaints companies actually paid. One reading is that history carries information a single complaint does not, though this comparison cannot separate that from the difference in the question asked, the number of labels or calibration.

Could a better prompt close the gap? On a random subset of complaints, the engines were asked a sharper question, whether the company will likely give money back, in three ways: on its own, with ten earlier complaints labeled paid or not paid, and with a short description of what tended to get paid in earlier years.

Table 5. Describing past outcomes in the question helps the engines a little, and examples do not close the gap to the classifier

The sharper question and the description together help a little, and rewording alone accounts for much of that. The labeled examples add little. None of the versions brings the engines level with the classifier. How much history does the classifier need? Refit on smaller samples of past complaints, it matches Jev at about a thousand.

Table 6. For money back, the classifier matches Jev with about a thousand past complaints

The classifier catches 49% with three hundred past complaints, 59% with a thousand and 66% with three thousand. That is far fewer labels than the review task needed, and a desk already has these records, because its own case system stores how each complaint closed.

These counts include calibration labels held out from each sample, and the complaints being scored are not used. They describe this classifier on these records, so treat them as a guide for a pilot. Whether a desk should build this model also depends on whether outcomes arrive in time, whether old records resemble today's complaints, and whether money back is what the desk wants to prioritize.

What these data cannot show

The routing score compares the engine with the consumer's own product choice, so some of what looks like engine error is consumer error, and these data cannot say how much. The money-back label is what the company did, which depends on the company as much as the complaint. The costs are assumptions, so the comparisons show how policies trade off under them and are not measured savings. The sample describes distinct narratives, not every filing, and money back is more common among them than among all filed complaints. The complaints are limited to narratives of at most 3,000 characters, which leaves out 8% of 2025 complaints with a narrative, because the public narratives were withdrawn in August 2026 and only a cut copy remains.

Tables

Table 1. A classifier trained on earlier complaints agrees with consumers' product choices about as often as Jev

Complaints whose predicted product matches the one the consumer selected, percent, by model, with 95% intervals

agree	conf_high	conf_low	kind	model	n
81.90	83.20	80.50	Decision engine	Jev 1.14.0	10,000
84.20	85.30	82.90	Decision engine	OpenAI Luna	10,000
84.40	85.50	83.30	General language model	Claude Haiku adapter	10,000
82.40	83.70	81.20	Trained classifier	TF-IDF + logistic	10,000

Note: Jev minus the classifier: −0.6 percentage points, from a bootstrap that resamples complaints within products.

Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026

Data: vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/agree-by-model.json

Table 2. At the same cutoff, Jev sends fewer complaints to people and misroutes more, because its stated probabilities run high

Where each routing policy sends complaints, percent of complaints, 2025, weighted

outcome	policy	share
Routed to the consumer's product	Jev, triage below the cutoff	73.40
Routed to another product	Jev, triage below the cutoff	10.20
Sent to a person for triage	Jev, triage below the cutoff	16.40
Routed to the consumer's product	Luna, triage below the cutoff	81.60
Routed to another product	Luna, triage below the cutoff	12.60
Sent to a person for triage	Luna, triage below the cutoff	5.90
Routed to the consumer's product	TF-IDF classifier, triage below the cutoff	53.70
Routed to another product	TF-IDF classifier, triage below the cutoff	3.80
Sent to a person for triage	TF-IDF classifier, triage below the cutoff	42.50
Routed to the consumer's product	Jev recalibrated, triage below the cutoff	60.40
Routed to another product	Jev recalibrated, triage below the cutoff	4.60
Sent to a person for triage	Jev recalibrated, triage below the cutoff	35.10
Routed to the consumer's product	TF-IDF recalibrated, triage below the cutoff	55
Routed to another product	TF-IDF recalibrated, triage below the cutoff	4
Sent to a person for triage	TF-IDF recalibrated, triage below the cutoff	41
Routed to the consumer's product	Jev, route everything	81.80
Routed to another product	Jev, route everything	18.10
Sent to a person for triage	Jev, route everything	0
Routed to the consumer's product	Haiku adapter, route everything (no probabilities)	84.40
Routed to another product	Haiku adapter, route everything (no probabilities)	15.60
Sent to a person for triage	Haiku adapter, route everything (no probabilities)	0
Routed to the consumer's product	Send everything to triage	0
Routed to another product	Send everything to triage	0
Sent to a person for triage	Send everything to triage	100

Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026

Data: vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/where-complaints-go.json

Table 3. For the same labels, recalibrating Jev costs less than fitting a classifier at every size tried

Expected routing cost per thousand complaints against local labels used, weighted; whiskers run from the lowest to the highest draw

conf_high	conf_low	cost	labels	model	n_labels
11720	4815	6289	50 labels	Classifier, fit and calibrated on these labels	50
4395	3471	3899	50 labels	Jev, recalibrated on these labels	50
6643	4417	5303	100 labels	Classifier, fit and calibrated on these labels	100
5308	3594	4128	100 labels	Jev, recalibrated on these labels	100
6000	4733	5125	200 labels	Classifier, fit and calibrated on these labels	200
4556	3471	3868	200 labels	Jev, recalibrated on these labels	200
5851	4468	4792	500 labels	Classifier, fit and calibrated on these labels	500
3769	3532	3613	500 labels	Jev, recalibrated on these labels	500
5187	4272	4500	1,000 labels	Classifier, fit and calibrated on these labels	1,000
4054	3471	3596	1,000 labels	Jev, recalibrated on these labels	1,000
4497	4215	4326	2,000 labels	Classifier, fit and calibrated on these labels	2,000
3839	3471	3587	2,000 labels	Jev, recalibrated on these labels	2,000
4208	4026	4119	5,000 labels	Classifier, fit and calibrated on these labels	5,000
3594	3471	3528	5,000 labels	Jev, recalibrated on these labels	5,000
4145	3889	3993	10,000 labels	Classifier, fit and calibrated on these labels	10,000
3594	3471	3546	10,000 labels	Jev, recalibrated on these labels	10,000
3837	3683	3742	30,000 labels	Classifier, fit and calibrated on these labels	30,000
n/a	n/a	3542	30,000 labels	Jev, recalibrated on these labels	30,000

Note: Platt scaling in place of isotonic calibration gives the same reading.

Source: CFPB Consumer Complaint Database; labels from August 2023 to 2024; policy scored on 2025 complaints, weighted; engine calls October 2026; assumed costs

Data: vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/cost-by-labels.json

Table 4. A classifier trained on past outcomes finds money-back complaints better than the models that only read the complaint

Share of complaints closed with monetary relief that fall in each model's top tenth by P(money back), percent

caught	kind	model
58.20	Decision engine	Jev 1.14.0
63.10	Decision engine	OpenAI Luna
51.80	General language model	Claude Haiku adapter
73.10	Trained classifier	TF-IDF + logistic

Note: Ranked over all complaints, the classifier's AUC is 0.936 and Jev's is 0.895; Jev minus the classifier is −0.041.

Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026

Data: vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-caught.json

Table 5. Describing past outcomes in the question helps the engines a little, and examples do not close the gap to the classifier

AUC for complaints closed with monetary relief, by model and how the question was posed, with 95% intervals; 2,000 sampled 2025 complaints

auc	conf_high	conf_low	group	label	n
0.947	0.959	0.932	Classifier trained on past outcomes	Classifier	2,000
0.881	0.912	0.847	Jev	Jev: original question	2,000
0.899	0.922	0.873	Jev	Jev: reworded to predict payment	2,000
0.903	0.926	0.878	Jev	Jev: reworded + ten labeled examples	2,000
0.913	0.931	0.891	Jev	Jev: reworded + outcome criteria	2,000
0.887	0.920	0.845	Luna	Luna: original question	2,000
0.903	0.929	0.875	Luna	Luna: reworded to predict payment	2,000
0.889	0.911	0.866	Luna	Luna: reworded + ten labeled examples	2,000
0.915	0.934	0.891	Luna	Luna: reworded + outcome criteria	2,000

Note: Only about one complaint in forty ended in money back, so the intervals are wide.

Source: CFPB Consumer Complaint Database, 2025 complaints; 2,000 drawn at random from the sample, weighted; engine calls October 2026

Data: vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-prompts.json

Table 6. For money back, the classifier matches Jev with about a thousand past complaints

Money-back complaints caught in the top tenth, percent, against past complaints used to train the classifier, weighted

caught	conf_high	conf_low	labels
34	51.90	11.20	100 complaints
49.30	59.50	30.50	300 complaints
58.80	65.30	53.10	1,000 complaints
66.40	69.10	63.60	3,000 complaints
69.90	72.90	66.60	10,000 complaints
74.90	75.90	74.20	30,000 complaints

Source: CFPB Consumer Complaint Database, 2025 distinct narratives with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026; ten refits per size, three at the largest

Data: vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/outcomes-to-catch-up.json

All estimates

Every fact in facts.json (73), sorted by id. The same values as JSON: facts.json.

Fact id	Value	Unit	Estimand	Population	Period	Interval	n	Weights	Source	Note
agree_diff_jev_tfidf	−0.6	percentage points	jev minus tfidf	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI −1.9 to 0.7	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
agree_jev	81.9%	percent of complaints	top product equals the consumer-selected product	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 80.5% to 83.2%	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
agree_jev_equal_products	75.2%	percent of complaints, each product counted equally	mean agreement across the eleven products	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
agree_tfidf	82.4%	percent of complaints	top product equals the consumer-selected product	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 81.2% to 83.7%	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
agree_tfidf_equal_products	63.6%	percent of complaints, each product counted equally	mean agreement across the eleven products	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
agree_tfidf_unweighted_fit	78.2%	percent of complaints	agreement for a classifier fit without weights	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
auc_money_diff_jev_tfidf	−0.041	difference in AUC	Jev minus TF-IDF	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI −0.054 to −0.029	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
auc_money_jev	0.895	area under the ROC curve	P(money back) against closed with monetary relief	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.881 to 0.908	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
auc_money_product_only	0.906	area under the ROC curve	ranking complaints by their product's relief rate in the training years	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
auc_money_tfidf	0.936	area under the ROC curve	P(money back) against closed with monetary relief	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.928 to 0.945	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
auc_within_adapter	0.730	area under the ROC curve	mean across products of the AUC within each product that has both outcomes	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.704 to 0.756	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
auc_within_diff_tfidf_jev	+0.027	difference in AUC	classifier minus Jev, within product	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI −0.009 to 0.065	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
auc_within_jev	0.742	area under the ROC curve	mean across products of the AUC within each product that has both outcomes	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.709 to 0.770	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
auc_within_luna	0.743	area under the ROC curve	mean across products of the AUC within each product that has both outcomes	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.706 to 0.772	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
auc_within_tfidf	0.769	area under the ROC curve	mean across products of the AUC within each product that has both outcomes	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.736 to 0.802	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
capture10_diff_tfidf_jev	+14.9	percentage points	classifier minus Jev	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 8.9 to 20.4	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
capture10_jev	58.2%	percent of complaints closed with monetary relief	share caught by fast-tracking the tenth of complaints with the highest P(money back)	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 53.0% to 63.4%		weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
capture10_tfidf	73.1%	percent of complaints closed with monetary relief	share caught by fast-tracking the tenth of complaints with the highest P(money back)	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 68.4% to 77.9%		weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
cost1k_jev	$0.05	US dollars per 1,000 complaints	mean billed cost per call, list price	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
lc_capture_1000	58.8%	percent of complaints closed with monetary relief	mean top-tenth capture over refits on 1,000 labeled complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
lc_capture_300	49.3%	percent of complaints closed with monetary relief	mean top-tenth capture over refits on 300 labeled complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
lc_capture_3000	66.4%	percent of complaints closed with monetary relief	mean top-tenth capture over refits on 3,000 labeled complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
loss_diff_jev_cal	−$195	US dollars per 1,000 complaints	jev_cal minus the classifier with a cutoff	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI −$419 to $27	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
loss_diff_jev_cut	+$368	US dollars per 1,000 complaints	jev_cut minus the classifier with a cutoff	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $97 to $655	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
loss_jev_all	$5,442	US dollars per 1,000 complaints	expected misroute and triage cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $5,041 to $5,861	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed costs; measured error mix against the consumer's product.
loss_jev_cal	$3,471	US dollars per 1,000 complaints	expected misroute and triage cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $3,243 to $3,709	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed costs; measured error mix against the consumer's product.
loss_jev_cut	$4,054	US dollars per 1,000 complaints	expected misroute and triage cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $3,740 to $4,384	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed costs; measured error mix against the consumer's product.
loss_tfidf_cal	$3,666	US dollars per 1,000 complaints	expected misroute and triage cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $3,466 to $3,885	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed costs; measured error mix against the consumer's product.
loss_tfidf_cut	$3,685	US dollars per 1,000 complaints	expected misroute and triage cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $3,482 to $3,893	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed costs; measured error mix against the consumer's product.
loss_triage_all	$6,000	US dollars per 1,000 complaints	expected misroute and triage cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $6,000 to $6,000	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed costs; measured error mix against the consumer's product.
lv_clf_500	$4,792	US dollars per thousand complaints	expected misroute and triage cost at the cutoff, mean across label draws	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed costs. Both arms use the same labeled draws.
lv_clf_full	$3,742	US dollars per thousand complaints	expected misroute and triage cost at the cutoff, mean across label draws	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed costs. Both arms use the same labeled draws.
lv_jev_500	$3,613	US dollars per thousand complaints	expected misroute and triage cost at the cutoff, mean across label draws	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed costs. Both arms use the same labeled draws.
lv_jev_full	$3,542	US dollars per thousand complaints	expected misroute and triage cost at the cutoff, mean across label draws	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed costs. Both arms use the same labeled draws.
lv_raw_jev	$4,054	US dollars per thousand complaints	raw Jev, no local fitting	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters; labels from August 2023 to 2024 complaints	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
mis_jev_cal	4.6%	percent of complaints	routed to a product other than the consumer's	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
mis_jev_cut	10.2%	percent of complaints	routed to a product other than the consumer's	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
mis_tfidf_cal	4.0%	percent of complaints	routed to a product other than the consumer's	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
mis_tfidf_cut	3.8%	percent of complaints	routed to a product other than the consumer's	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
misroute_cost	$30	US dollars per misrouted complaint	assumed cost: three days of delay at $10	the complaint desk's routing decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	CFPB Consumer Complaint Database
money_extra	$120	US dollars per misrouted complaint that ends in money back	assumed extra cost	the complaint desk's routing decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	CFPB Consumer Complaint Database
n_complaints	10,000	complaints	sample size	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
n_population	519,841	complaints	2025 complaints within the length limit that the sample stands for	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
narrative_cap	3,000	characters	length cap of the stored narratives	CFPB narratives in the July 2026 copy	2025			not applicable	CFPB Consumer Complaint Database
queue_diff_value_cost	+79	per 1,000 complaints	value minus reviewing the least certain, cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI −56 to 233	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
queue_diff_value_left_money	−0.43	per 1,000 complaints	value minus reviewing the least certain, left_money	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI −0.91 to 0.09	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
queue_diff_value_per_minute_cost	−425	per 1,000 complaints	value_per_minute minus reviewing the least certain, cost	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI −639 to −198	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
queue_diff_value_per_minute_left	−12.15	per 1,000 complaints	value_per_minute minus reviewing the least certain, left	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI −18.81 to −4.95	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
queue_minutes	400	reviewer minutes per 1,000 complaints	assumed review budget	the complaint desk's routing decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	CFPB Consumer Complaint Database
queue_none_cost	$5,827	US dollars per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $5,402 to $6,277	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_none_left	181.5	misroutes left per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 168.2 to 195.6	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_none_left_money	3.18	misrouted money-back complaints left per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 2.26 to 4.20	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_none_reviewed	0.0%	percent of complaints reviewed	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026		10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_uncertain_cost	$3,499	US dollars per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $3,144 to $3,852	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_uncertain_left	110.4	misroutes left per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 99.4 to 121.6	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_uncertain_left_money	1.57	misrouted money-back complaints left per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.89 to 2.31	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_uncertain_reviewed	14.3%	percent of complaints reviewed	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 13.1% to 15.6%	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_value_cost	$3,579	US dollars per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $3,229 to $3,944	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_value_left	114.7	misroutes left per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 103.6 to 126.1	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_value_left_money	1.14	misrouted money-back complaints left per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.54 to 1.88	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_value_per_minute_cost	$3,074	US dollars per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI $2,732 to $3,429	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_value_per_minute_left	98.2	misroutes left per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 87.0 to 109.5	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_value_per_minute_left_money	1.06	misrouted money-back complaints left per 1,000 complaints	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 0.50 to 1.73	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_value_per_minute_reviewed	20.2%	percent of complaints reviewed	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 18.8% to 21.6%	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
queue_value_reviewed	14.1%	percent of complaints reviewed	review queue of 400 reviewer minutes per 1,000 complaints	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026	95% CI 13.0% to 15.3%	10,000 complaints	weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database	Assumed review time, reviewer accuracy and costs; measured misroutes against the consumer's product.
route_cutoff	0.8	probability that the top product is right	1 minus triage cost over misroute cost	the complaint desk's routing decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	CFPB Consumer Complaint Database
share_relief	2.6%	percent of complaints	company response is closed with monetary relief	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
share_too_long	7.7%	percent of deduplicated 2025 complaints with a narrative	narratives over 3,000 characters, excluded	deduplicated 2025 CFPB complaints with a narrative	2025			full count	CFPB Consumer Complaint Database
tri_jev_cal	35.1%	percent of complaints	sent to triage	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
tri_jev_cut	16.4%	percent of complaints	sent to triage	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
tri_tfidf_cal	41.0%	percent of complaints	sent to triage	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
tri_tfidf_cut	42.5%	percent of complaints	sent to triage	10,000 sampled 2025 CFPB complaints with a narrative of at most 3,000 characters	complaints received in 2025; engines called October 2026			weighted to the product mix of those complaints (sample_weight)	CFPB Consumer Complaint Database
triage_cost	$6	US dollars per complaint sent to triage	assumed cost: four minutes of a triage handler	the complaint desk's routing decision (assumed inputs)	assumed inputs, October 2026			not applicable; assumed inputs	CFPB Consumer Complaint Database

Definitions

The chapter marks no glossary term.

Methods

Two decision engines and a general language model read each sampled complaint narrative and answered two typed questions: which product it is about, and whether the consumer describes money the company could return. A classifier trained on earlier complaints answered the same two. Nothing was rated by hand.

Which complaints

CFPB withdrew complaint narratives from public access in August 2026, so the only copy available keeps the first 3,000 characters of each narrative. The sample keeps only complaints whose whole narrative fits, which excludes 7.7% of 2025 complaints with a narrative. Every result describes complaints within that length.

The sample holds the same number of complaints for each of the eleven products, and weights restore each product's share of 2025 complaints.

Labels

Routing is scored against the product the consumer selected when filing. Consumers sometimes pick the wrong product, so a disagreement can be the consumer's mistake rather than the engine's.

The money-back question is scored against the company's recorded response, closed with monetary relief. That outcome depends on the company as well as the complaint.

Intervals

Intervals are 95% bootstrap intervals that resample complaints within product strata, keep each product's share fixed and use the same resamples for every policy. They describe sampling error for 2025 complaints and say nothing about other years or other companies.

What these data cannot show

References

The chapter cites no reference.

Provenance

Build script: decision_ai/chapters/resolving-and-routing-customer-complaints/build.py (sha256 ad8930e6ff310f8e48d53f32dc53dc37c1156cc14259c5adbe4b587e8d5a291f)
Built on: 2026-10-29
Site commit: 252942a900a49d01fe9f278e18459eb87b20b4a0

Outputs

File	SHA-256
facts.json	20c650ec38ef9810222843d6e40613ef2d6545f645f749ed57035e7fe6d7ea92
figures/agree-by-model.json	1b7f77c56eee87f3469be5ac6cf9affb228c657e53801d78d2e3366e9a7fc1f8
figures/cost-by-labels.json	34360a16ffe4d9b7133cacf6e24209e9e0410bcb594a1d42581a2fab26e5a5ce
figures/money-back-caught.json	03c660401c1437e49780d3a291312f0aa46f0a7debdc69855fd6a151018ea1e8
figures/money-back-prompts.json	0d8296a19c6854ea1f9c9ef1851b6e05b607916c3ef3ad6e632a3a2c0e89d754
figures/outcomes-to-catch-up.json	3a66877ee2c772f4622dda2511a750923589ebfad0c9b74ee85deca531a3aa95
figures/where-complaints-go.json	43adef141cfb6ed2a770238c094ba0436f6ea8ca0c0f591d0e1d38a6b6d305b8
meta.json	e7892b08742ced88d370d9d8660f15d3481e33f08529e79bc4f5923f2673ef90
public/data/resolving-and-routing-customer-complaints/flow.json	85255672e8768ac873e89cd12604fb38c55cd4c0f5042805034be293c2a785ef
public/data/resolving-and-routing-customer-complaints/items.json	27123950e0178d495717b23b6f360caf78b7babd3f5f88f989218637fe0dd855
public/data/resolving-and-routing-customer-complaints/policy.json	c2c0475becd99606c6f49869aeeaece194b993c52165a170c0114b6bec2ca6b5
public/data/resolving-and-routing-customer-complaints/queue.json	98f9a4750c6905d6d2c87450977d629f97d0a14659f845a043eca707a0b9bc4c
refs.json	ca3d163bab055381827226140568f3bef7eaac187cebd76878e0b63e9e442356

Machine formats

Report (Markdown): https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints/report.md
Every estimate (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/facts.json
Table 1, figure "agree-by-model" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/agree-by-model.json
Table 2, figure "where-complaints-go" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/where-complaints-go.json
Table 3, figure "cost-by-labels" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/cost-by-labels.json
Table 4, figure "money-back-caught" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-caught.json
Table 5, figure "money-back-prompts" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/money-back-prompts.json
Table 6, figure "outcomes-to-catch-up" (JSON): https://vishalsingh.org/decisionai/report/resolving-and-routing-customer-complaints/figures/outcomes-to-catch-up.json
Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt