Chapter 2.1

Resolving and routing customer complaints

A classifier trained on earlier complaints matches consumers' product choices about as often as Jev once it is fit to the actual mix, and it finds more of the complaints that end in money back, where it passes Jev with far fewer labels than the review task needs.

Author: Vishal Singh
Affiliation: NYU Stern
Published: 29 October 2026

Full report: every estimate with its interval, and every figure as a table.

DraftFigures and numbers may change as the analysis is rebuilt.what is still being checked

First draft, redesigned without human rating: routing is scored against the consumer's own product choice, and the money-back question against the company's recorded response.

A bank's complaint desk gets a steady stream of written complaints, and each one needs to reach the right team. Send a mortgage complaint to the credit card team and it sits for days before someone sends it back. This chapter asks a decision engine to read each complaint and pick the product it is about, then asks the question every desk manager asks next: when should a person check the engine's pick before it goes out?

The complaints are 10,000 sampled from distinct narratives filed with the Consumer Financial Protection Bureau in 2025, standing for 519,841 narratives; repeated narratives were removed, which matters because many credit-reporting complaints use the same text. No one rated them by hand. Instead, each complaint carries two answers that already exist: the product the consumer selected when filing, and what the company eventually did about it.

Jev and a classifier fit to the actual complaint mix match the consumer's product choice about equally often, 82% and 82%, and once both are recalibrated the cost comparison shows no clear winner at the assumed costs. So the choice between them rests on what a desk can run and check, and either one still needs people for uncertain complaints. Consumer choices are sometimes wrong, and the costs and review times are assumptions. The chapter follows the desk from automatic routing to human review, a limit on review time and how to spend it, then takes up one companion question: which complaints end in money back.

The decision

Owner and actions: The complaint desk lead decides where each complaint goes; operations owns the cost of delay.
Evidence at decision time: The complaint narrative as the consumer wrote it, and nothing added later.
Simplest alternative: A word-counting classifier trained on complaints from earlier years.
What establishes correctness: The consumer's own product choice, which is sometimes wrong, and the company's recorded response.
When the model is wrong or cannot decide: A misrouted complaint waits days for the right team; a person reviewing every complaint costs staff time.
Does the policy pay: Expected cost per thousand complaints under stated costs for misroutes and triage.

Picking the product

Each engine saw the narrative and a list of eleven products with a short definition of each, and picked one. Jev and Luna also returned a probability for every product; the Haiku adapter returned only its pick. The chart compares how often the top pick matches what the consumer chose.

Figure 1

A classifier trained on earlier complaints agrees with consumers' product choices about as often as Jev

Complaints whose predicted product matches the one the consumer selected, percent, by model, with 95% intervals

Decision engine
General language model
Trained classifier

Show data

Data for A classifier trained on earlier complaints agrees with consumers' product choices about as often as Jev
agree	conf_high	conf_low	kind	model	n
81.90	83.20	80.50	Decision engine	Jev 1.14.0	10,000
84.20	85.30	82.90	Decision engine	OpenAI Luna	10,000
84.40	85.50	83.30	General language model	Claude Haiku adapter	10,000
82.40	83.70	81.20	Trained classifier	TF-IDF + logistic	10,000

Each dot is one model, and the whisker is its 95% interval. Agreement is not the same as being right, because some consumers pick the wrong product. Counted equally across products, Jev leads the classifier (75.2% against 63.6%), because the classifier was fit to the actual mix.Note: Jev minus the classifier: −0.6 percentage points, from a bootstrap that resamples complaints within products.Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026

Jev agrees on 82% of complaints and the classifier on 82%, a gap of −0.6 (95% CI −1.9 to 0.7) points. How the classifier is trained decides this comparison. The training sample holds equal numbers of each product, and fitting it without weights drops the classifier to 78%, while the weighted fit does poorly on rare products: counted equally across products, Jev scores 75% and the classifier 64%. Jev costs about $0.05 per thousand complaints.

Agreement is a soft score here. Consumers choose their own product from a form, and some choose badly: a complaint about a debt collector chasing a credit card balance can honestly belong to either team. The flows below show where Jev and the consumers part ways. The largest disagreements run between credit reporting and debt collection, two products that often appear in the same complaint.

Routing flows

Most complaints land in the product the consumer chose; the largest other flows run between credit reporting and debt collection

Weighted flows from consumer-selected product to Jev's predicted product, 2025 complaints

Same category (solid)
Another category (hatched)

Select a category (click, or Tab to it and press Enter) to highlight its flows; Escape or a second press clears.

81.9% of 9,998 items (weighted) are predicted in their recorded category by Jev 1.14.0. The largest flow to another category is Debt collection to Credit reporting, 2.8% of all items.

Show data

Data for Most complaints land in the product the consumer chose; the largest other flows run between credit reporting and debt collection
Consumer-selected product	Jev's predicted product	Same category	Share of all items, weighted (%)	Share of the consumer-selected product row, weighted (%)	Items (unweighted)
Checking or savings	Checking or savings	yes	6.9	80.7	734
Checking or savings	Credit card	no	0.6	6.5	59
Checking or savings	Credit reporting	no	0.1	1.0	9
Checking or savings	Debt collection	no	0.0	0.6	5
Checking or savings	Money transfer	no	0.7	8.8	80
Checking or savings	Mortgage	no	0.0	0.1	1
Checking or savings	Payday or personal loan	no	0.1	0.7	6
Checking or savings	Prepaid card	no	0.1	1.3	12
Checking or savings	Vehicle loan or lease	no	0.0	0.3	3
Credit card	Checking or savings	no	0.2	2.9	26
Credit card	Credit card	yes	5.9	84.4	766
Credit card	Credit reporting	no	0.5	6.5	59
Credit card	Debt collection	no	0.2	2.4	22
Credit card	Debt or credit management	no	0.0	0.2	2
Credit card	Money transfer	no	0.1	1.4	13
Credit card	Mortgage	no	0.0	0.1	1
Credit card	Payday or personal loan	no	0.1	1.1	10
Credit card	Prepaid card	no	0.1	1.0	9
Credit reporting	Checking or savings	no	0.4	0.7	6
Credit reporting	Credit card	no	1.7	3.1	28
Credit reporting	Credit reporting	yes	46.6	84.5	769
Credit reporting	Debt collection	no	2.7	4.8	44
Credit reporting	Debt or credit management	no	0.1	0.1	1
Credit reporting	Money transfer	no	0.1	0.2	2
Credit reporting	Mortgage	no	0.4	0.7	6
Credit reporting	Payday or personal loan	no	0.5	1.0	9
Credit reporting	Student loan	no	1.4	2.5	23
Credit reporting	Vehicle loan or lease	no	1.3	2.4	22
Debt collection	Checking or savings	no	0.0	0.2	2
Debt collection	Credit card	no	0.7	5.1	46
Debt collection	Credit reporting	no	2.8	20.6	187
Debt collection	Debt collection	yes	9.5	70.3	639
Debt collection	Debt or credit management	no	0.0	0.3	3
Debt collection	Money transfer	no	0.0	0.2	2
Debt collection	Mortgage	no	0.0	0.3	3
Debt collection	Payday or personal loan	no	0.2	1.2	11
Debt collection	Student loan	no	0.1	0.4	4
Debt collection	Vehicle loan or lease	no	0.2	1.3	12
Debt or credit management	Checking or savings	no	0.0	3.6	33
Debt or credit management	Credit card	no	0.0	8.5	78
Debt or credit management	Credit reporting	no	0.1	20.9	190
Debt or credit management	Debt collection	no	0.1	21.1	191
Debt or credit management	Debt or credit management	yes	0.2	33.0	300
Debt or credit management	Money transfer	no	0.0	2.1	20
Debt or credit management	Mortgage	no	0.0	2.8	26
Debt or credit management	Payday or personal loan	no	0.0	3.8	35
Debt or credit management	Prepaid card	no	0.0	0.6	5
Debt or credit management	Student loan	no	0.0	1.5	13
Debt or credit management	Vehicle loan or lease	no	0.0	1.9	18
Money transfer	Checking or savings	no	0.7	9.9	90
Money transfer	Credit card	no	0.2	2.8	25
Money transfer	Credit reporting	no	0.0	0.2	2
Money transfer	Debt collection	no	0.0	0.1	1
Money transfer	Debt or credit management	no	0.0	0.1	1
Money transfer	Money transfer	yes	6.0	85.6	777
Money transfer	Payday or personal loan	no	0.0	0.5	5
Money transfer	Prepaid card	no	0.0	0.4	4
Money transfer	Vehicle loan or lease	no	0.0	0.3	3
Mortgage	Checking or savings	no	0.0	0.9	8
Mortgage	Credit card	no	0.0	0.6	6
Mortgage	Credit reporting	no	0.0	1.0	9
Mortgage	Debt collection	no	0.0	1.4	13
Mortgage	Debt or credit management	no	0.0	0.1	1
Mortgage	Money transfer	no	0.0	0.2	2
Mortgage	Mortgage	yes	2.2	93.0	845
Mortgage	Payday or personal loan	no	0.1	2.2	20
Mortgage	Student loan	no	0.0	0.1	1
Mortgage	Vehicle loan or lease	no	0.0	0.4	4
Payday or personal loan	Checking or savings	no	0.0	2.5	23
Payday or personal loan	Credit card	no	0.1	5.3	48
Payday or personal loan	Credit reporting	no	0.1	5.9	54
Payday or personal loan	Debt collection	no	0.1	9.7	88
Payday or personal loan	Debt or credit management	no	0.0	0.6	6
Payday or personal loan	Money transfer	no	0.0	0.6	6
Payday or personal loan	Mortgage	no	0.0	1.8	17
Payday or personal loan	Payday or personal loan	yes	1.0	68.6	623
Payday or personal loan	Prepaid card	no	0.0	0.1	1
Payday or personal loan	Student loan	no	0.0	0.8	7
Payday or personal loan	Vehicle loan or lease	no	0.1	4.0	36
Prepaid card	Checking or savings	no	0.1	12.0	109
Prepaid card	Credit card	no	0.1	10.6	96
Prepaid card	Credit reporting	no	0.0	0.3	3
Prepaid card	Debt collection	no	0.0	0.7	6
Prepaid card	Money transfer	no	0.1	16.8	153
Prepaid card	Mortgage	no	0.0	0.3	2
Prepaid card	Payday or personal loan	no	0.0	0.7	7
Prepaid card	Prepaid card	yes	0.4	58.6	533
Student loan	Checking or savings	no	0.0	0.1	1
Student loan	Credit card	no	0.0	0.9	8
Student loan	Credit reporting	no	0.1	6.4	58
Student loan	Debt collection	no	0.1	2.8	25
Student loan	Money transfer	no	0.0	0.1	1
Student loan	Mortgage	no	0.0	0.2	2
Student loan	Payday or personal loan	no	0.0	2.4	22
Student loan	Prepaid card	no	0.0	0.2	2
Student loan	Student loan	yes	1.7	86.9	790
Vehicle loan or lease	Checking or savings	no	0.0	0.5	5
Vehicle loan or lease	Credit card	no	0.1	2.6	24
Vehicle loan or lease	Credit reporting	no	0.2	7.6	69
Vehicle loan or lease	Debt collection	no	0.1	3.5	32
Vehicle loan or lease	Debt or credit management	no	0.0	0.1	1
Vehicle loan or lease	Money transfer	no	0.0	0.1	1
Vehicle loan or lease	Mortgage	no	0.0	0.4	4
Vehicle loan or lease	Payday or personal loan	no	0.1	3.4	31
Vehicle loan or lease	Prepaid card	no	0.0	0.1	1
Vehicle loan or lease	Student loan	no	0.0	0.2	2
Vehicle loan or lease	Vehicle loan or lease	yes	1.7	81.3	739

Follow the bands that change product. Some are engine errors and some are consumers who picked the wrong product; these data cannot tell which.Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted

When to ask a person first

Suppose a misrouted complaint costs $30 in delay and sending a complaint to a person for triage costs $6 of staff time. Routing pays when the chance the pick is right is high enough that the expected misroute cost falls below the triage cost. At these costs the cutoff is 0.8: route when the model is more sure than that, and send everything else to a person.

Triage band

Automate, review or triage

Loading data

The next chart shows what each policy does with the whole stream of complaints.

Figure 2

At the same cutoff, Jev sends fewer complaints to people and misroutes more, because its stated probabilities run high

Where each routing policy sends complaints, percent of complaints, 2025, weighted

Routed to the consumer's product
Routed to another product
Sent to a person for triage

Show data

Data for At the same cutoff, Jev sends fewer complaints to people and misroutes more, because its stated probabilities run high
outcome	policy	share
Routed to the consumer's product	Jev, triage below the cutoff	73.40
Routed to another product	Jev, triage below the cutoff	10.20
Sent to a person for triage	Jev, triage below the cutoff	16.40
Routed to the consumer's product	Luna, triage below the cutoff	81.60
Routed to another product	Luna, triage below the cutoff	12.60
Sent to a person for triage	Luna, triage below the cutoff	5.90
Routed to the consumer's product	TF-IDF classifier, triage below the cutoff	53.70
Routed to another product	TF-IDF classifier, triage below the cutoff	3.80
Sent to a person for triage	TF-IDF classifier, triage below the cutoff	42.50
Routed to the consumer's product	Jev recalibrated, triage below the cutoff	60.40
Routed to another product	Jev recalibrated, triage below the cutoff	4.60
Sent to a person for triage	Jev recalibrated, triage below the cutoff	35.10
Routed to the consumer's product	TF-IDF recalibrated, triage below the cutoff	55
Routed to another product	TF-IDF recalibrated, triage below the cutoff	4
Sent to a person for triage	TF-IDF recalibrated, triage below the cutoff	41
Routed to the consumer's product	Jev, route everything	81.80
Routed to another product	Jev, route everything	18.10
Sent to a person for triage	Jev, route everything	0
Routed to the consumer's product	Haiku adapter, route everything (no probabilities)	84.40
Routed to another product	Haiku adapter, route everything (no probabilities)	15.60
Sent to a person for triage	Haiku adapter, route everything (no probabilities)	0
Routed to the consumer's product	Send everything to triage	0
Routed to another product	Send everything to triage	0
Sent to a person for triage	Send everything to triage	100

Read each bar from left to right: complaints routed to the product the consumer chose, routed somewhere else, and sent to a person. Policies with a cutoff send their least certain complaints to a person.Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026

With the cutoff taken at face value, Jev sends 16% of complaints to a person and misroutes 10%, and the classifier sends 43% and misroutes 4%. Priced at the assumed costs, that is $4,054 per thousand complaints for Jev and $3,685 for the classifier. Routing everything by Jev with no check costs $5,442, and sending everything to a person costs $6,000.

The gap of +$368 (95% CI $97 to $655) is about calibration more than about reading. Jev's stated probabilities run high, so it routes complaints it should send to a person. Recalibrate both models on other folds of the sample and Jev costs $3,471, sending 35% to a person, against $3,666 for the classifier.

Policy comparison

With a cutoff, every model sends a share of complaints to people and misroutes fewer

Expected cost per 1,000 complaints, misroute and triage shares by routing policy, with 95% intervals; 2025 complaints, weighted

Policy	derived	measured	measured
Jev, triage below the cutoff (chapter policy)	$4,054 ($3,740–$4,384)	10.2% (9.2%–11.3%)	16.4% (15.1%–17.8%)
Luna, triage below the cutoff	$4,127 ($3,797–$4,457)	12.6% (11.5%–13.7%)	5.9% (5.1%–6.7%)
TF-IDF classifier, triage below the cutoff	$3,685 ($3,482–$3,893)	3.8% (3.1%–4.4%)	42.5% (40.8%–44.2%)
Jev recalibrated, triage below the cutoff	$3,471 ($3,243–$3,709)	4.6% (3.8%–5.3%)	35.1% (33.4%–36.9%)
TF-IDF recalibrated, triage below the cutoff	$3,666 ($3,466–$3,885)	4.0% (3.4%–4.7%)	41.0% (39.4%–42.7%)
Jev, route everything	$5,442 ($5,041–$5,861)	18.1% (16.8%–19.5%)	0.0% (0.0%–0.0%)
Haiku adapter, route everything (no probabilities)	$4,683 ($4,354–$5,011)	15.6% (14.5%–16.7%)	0.0% (0.0%–0.0%)
Send everything to triage	$6,000 ($6,000–$6,000)	0.0% (0.0%–0.0%)	100.0% (100.0%–100.0%)

n = 10,000 items. Intervals are paired bootstrap 95% intervals from the pipeline.

Jev, triage below the cutoff: expected cost per 1,000 $4,054 (95% interval $3,740 to $4,384). Jev recalibrated, triage below the cutoff has the lowest point estimate on this measure, $3,471 (95% interval $3,243 to $3,709).

Show data

Data for With a cutoff, every model sends a share of complaints to people and misroutes fewer
Policy	Expected cost per 1,000	Lower 95%	Upper 95%	Misrouted	Lower 95%	Upper 95%	Sent to triage	Lower 95%	Upper 95%
Jev, triage below the cutoff	4054	3740	4384	10.2	9.2	11.3	16.4	15.1	17.8
Luna, triage below the cutoff	4127	3797	4457	12.6	11.5	13.7	5.9	5.1	6.7
TF-IDF classifier, triage below the cutoff	3685	3482	3893	3.8	3.1	4.4	42.5	40.8	44.2
Jev recalibrated, triage below the cutoff	3471	3243	3709	4.6	3.8	5.3	35.1	33.4	36.9
TF-IDF recalibrated, triage below the cutoff	3666	3466	3885	4.0	3.4	4.7	41.0	39.4	42.7
Jev, route everything	5442	5041	5861	18.1	16.8	19.5	0.0	0.0	0.0
Haiku adapter, route everything (no probabilities)	4683	4354	5011	15.6	14.5	16.7	0.0	0.0	0.0
Send everything to triage	6000	6000	6000	0.0	0.0	0.0	100.0	100.0	100.0

Read across a row: what a policy costs and where its cost comes from. Costs are assumed; misroutes are counted against the consumer's own product choice.Source: CFPB Consumer Complaint Database, 2025 complaints; 10,000 sampled, weighted; engine calls October 2026

That comparison gives both models plenty of labels. A desk starting out would have few, so the next chart gives both the same labeled complaints from earlier years.

Figure 3

For the same labels, recalibrating Jev costs less than fitting a classifier at every size tried

Expected routing cost per thousand complaints against local labels used, weighted; whiskers run from the lowest to the highest draw

Classifier, fit and calibrated on these labels
Jev, recalibrated on these labels

Show data

Data for For the same labels, recalibrating Jev costs less than fitting a classifier at every size tried
conf_high	conf_low	cost	labels	model	n_labels
11720	4815	6289	50 labels	Classifier, fit and calibrated on these labels	50
4395	3471	3899	50 labels	Jev, recalibrated on these labels	50
6643	4417	5303	100 labels	Classifier, fit and calibrated on these labels	100
5308	3594	4128	100 labels	Jev, recalibrated on these labels	100
6000	4733	5125	200 labels	Classifier, fit and calibrated on these labels	200
4556	3471	3868	200 labels	Jev, recalibrated on these labels	200
5851	4468	4792	500 labels	Classifier, fit and calibrated on these labels	500
3769	3532	3613	500 labels	Jev, recalibrated on these labels	500
5187	4272	4500	1,000 labels	Classifier, fit and calibrated on these labels	1,000
4054	3471	3596	1,000 labels	Jev, recalibrated on these labels	1,000
4497	4215	4326	2,000 labels	Classifier, fit and calibrated on these labels	2,000
3839	3471	3587	2,000 labels	Jev, recalibrated on these labels	2,000
4208	4026	4119	5,000 labels	Classifier, fit and calibrated on these labels	5,000
3594	3471	3528	5,000 labels	Jev, recalibrated on these labels	5,000
4145	3889	3993	10,000 labels	Classifier, fit and calibrated on these labels	10,000
3594	3471	3546	10,000 labels	Jev, recalibrated on these labels	10,000
3837	3683	3742	30,000 labels	Classifier, fit and calibrated on these labels	30,000
n/a	n/a	3542	30,000 labels	Jev, recalibrated on these labels	30,000

Further left is cheaper. Both arms use the same labeled complaints from earlier years, and the line marks raw Jev. Each count is every local label used, including the classifier's calibration labels; the complaints being scored are not among them. Whiskers are ranges across draws and not confidence intervals.Note: Platt scaling in place of isotonic calibration gives the same reading.Source: CFPB Consumer Complaint Database; labels from August 2023 to 2024; policy scored on 2025 complaints, weighted; engine calls October 2026; assumed costs

With five hundred labels, recalibrated Jev costs $3,600 per thousand complaints, against $4,100 for raw Jev and $4,800 for a classifier fit on those labels. With the full sample the classifier comes closer, at $3,700 against $3,500. Labels from earlier years are what a desk would really have, and these figures leave out what each model costs to run.

When reviewer time is fixed

So far a person has been free to read as many complaints as the cutoff sends. Suppose the desk instead has 400 reviewer minutes for every thousand complaints, and everything outside the queue goes where Jev's pick sends it. Which complaints should a person read? The complaints and the costs stay as they were; what changes is that the desk can no longer send a person everything below the cutoff.

Uncertainty is one reason to look at a complaint. Consequence is another, so the queue below also uses a second model. Its estimate of money back, from the companion section that follows, stands in for a misroute that matters more, under an assumed extra cost. It is a proxy and not a measured loss from a misroute. Read the columns separately, because review workload, misroutes, misrouted money-back complaints and assumed dollars can move in different directions.

Policy comparison

With reviewer time fixed, ranking by consequence and length changes which misroutes get caught

Outcomes per 1,000 complaints with 400 reviewer minutes per 1,000 complaints spent on Jev's routing picks, with 95% intervals; 2025 complaints, weighted

Policy	derived	measured	measured	derived
Review by expected loss per minute (chapter policy)	20.2% (18.8%–21.6%)	98.2 (87.0–109.5)	1.06 (0.50–1.73)	$3,074 ($2,732–$3,429)
No review	0.0% (0.0%–0.0%)	181.5 (168.2–195.6)	3.18 (2.26–4.20)	$5,827 ($5,402–$6,277)
Review the least certain picks	14.3% (13.1%–15.6%)	110.4 (99.4–121.6)	1.57 (0.89–2.31)	$3,499 ($3,144–$3,852)
Review by expected loss	14.1% (13.0%–15.3%)	114.7 (103.6–126.1)	1.14 (0.54–1.88)	$3,579 ($3,229–$3,944)

n = 10,000 items. Intervals are paired bootstrap 95% intervals from the pipeline.

Review by expected loss per minute: complaints reviewed 20.2% (95% interval 18.8% to 21.6%). It has the highest point estimate on this measure.

Show data

Data for With reviewer time fixed, ranking by consequence and length changes which misroutes get caught
Policy	Complaints reviewed	Lower 95%	Upper 95%	Misroutes left	Lower 95%	Upper 95%	Misrouted money-back complaints left	Lower 95%	Upper 95%	Expected cost	Lower 95%	Upper 95%
Review by expected loss per minute	20.2	18.8	21.6	98.2	87.0	109.5	1.06	0.50	1.73	3074	2732	3429
No review	0.0	0.0	0.0	181.5	168.2	195.6	3.18	2.26	4.20	5827	5402	6277
Review the least certain picks	14.3	13.1	15.6	110.4	99.4	121.6	1.57	0.89	2.31	3499	3144	3852
Review by expected loss	14.1	13.0	15.3	114.7	103.6	126.1	1.14	0.54	1.88	3579	3229	3944

Read down each column. Every queue spends the same reviewer minutes, and complaints outside it go where Jev's pick sends them. Review time, reviewer accuracy and costs are assumed; misroutes are counted against the consumer's product.Source: CFPB Consumer Complaint Database, 2025 distinct narratives; 10,000 sampled, weighted; engine calls October 2026; assumed review time and costs

Reviewing the least certain picks is the natural rule, and it leaves 110.4 misroutes per thousand complaints, down from 181.5 with no review. Adding the consequence moves attention toward complaints that may involve money, leaving 1.14 misrouted money-back complaints against 1.57, a difference of −0.43 (95% CI −0.91 to 0.09), whose range includes zero. Adding review time changes more: because short complaints take less time, the same minutes cover 20% of complaints instead of 14%, and the expected cost changes by −425 (95% CI −639 to −198) dollars per thousand. That gain rests on the assumed link between length and review time, and on a short complaint being reviewed as accurately as a long one. A desk would need to measure both before running this queue.

A companion question: which complaints end in money back

This is a different job from routing: it predicts what a company will do later, not what a complaint is about. A desk might fast-track complaints that are likely to end with the company paying the consumer back. Only 3% of these complaints closed with monetary relief, so finding them early matters. Each engine was asked whether the consumer describes money the company could return, and the classifier was trained directly on which earlier complaints ended in relief.

Figure 4

A classifier trained on past outcomes finds money-back complaints better than the models that only read the complaint

Share of complaints closed with monetary relief that fall in each model's top tenth by P(money back), percent

Decision engine
General language model
Trained classifier

Show data

Data for A classifier trained on past outcomes finds money-back complaints better than the models that only read the complaint
caught	kind	model
58.20	Decision engine	Jev 1.14.0
63.10	Decision engine	OpenAI Luna
51.80	General language model	Claude Haiku adapter
73.10	Trained classifier	TF-IDF + logistic

Each stem shows how many money-back complaints a desk would catch if it fast-tracked the tenth of complaints the model ranks highest. Picking a tenth at random would catch about a tenth of them.Note: Ranked over all complaints, the classifier's AUC is 0.936 and Jev's is 0.895; Jev minus the classifier is −0.041.Source: CFPB Consumer Complaint Database, 2025 complaints with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026

Fast-tracking the tenth of complaints the classifier ranks highest would catch 73% of the money-back cases, against 58% for Jev, a gap of +14.9 (95% CI 8.9 to 20.4) points. Much of any model's score here comes from knowing the product, because money back is far more common for some products than others: ranking complaints by their product's past rate alone gives an AUC of 0.906, above Jev's 0.895. Within a product, the classifier's lead narrows to 0.769 against 0.742, a difference of +0.027 (95% CI −0.009 to 0.065). The engine answers the question it was asked, whether the consumer describes a refundable loss. The classifier learned from history which complaints companies actually paid. One reading is that history carries information a single complaint does not, though this comparison cannot separate that from the difference in the question asked, the number of labels or calibration.

Could a better prompt close the gap? On a random subset of complaints, the engines were asked a sharper question, whether the company will likely give money back, in three ways: on its own, with ten earlier complaints labeled paid or not paid, and with a short description of what tended to get paid in earlier years.

Figure 5

Describing past outcomes in the question helps the engines a little, and examples do not close the gap to the classifier

AUC for complaints closed with monetary relief, by model and how the question was posed, with 95% intervals; 2,000 sampled 2025 complaints

Classifier trained on past outcomes
Jev
Luna

Show data

Data for Describing past outcomes in the question helps the engines a little, and examples do not close the gap to the classifier
auc	conf_high	conf_low	group	label	n
0.947	0.959	0.932	Classifier trained on past outcomes	Classifier	2,000
0.881	0.912	0.847	Jev	Jev: original question	2,000
0.899	0.922	0.873	Jev	Jev: reworded to predict payment	2,000
0.903	0.926	0.878	Jev	Jev: reworded + ten labeled examples	2,000
0.913	0.931	0.891	Jev	Jev: reworded + outcome criteria	2,000
0.887	0.920	0.845	Luna	Luna: original question	2,000
0.903	0.929	0.875	Luna	Luna: reworded to predict payment	2,000
0.889	0.911	0.866	Luna	Luna: reworded + ten labeled examples	2,000
0.915	0.934	0.891	Luna	Luna: reworded + outcome criteria	2,000

Each dot is one model with one version of the question. Moving right means the model ranks money-back complaints above the rest more often. The classifier learned from earlier outcomes; the engines saw only the question, plus ten labeled examples or a short description of what tended to get paid.Note: Only about one complaint in forty ended in money back, so the intervals are wide.Source: CFPB Consumer Complaint Database, 2025 complaints; 2,000 drawn at random from the sample, weighted; engine calls October 2026

The sharper question and the description together help a little, and rewording alone accounts for much of that. The labeled examples add little. None of the versions brings the engines level with the classifier. How much history does the classifier need? Refit on smaller samples of past complaints, it matches Jev at about a thousand.

Figure 6

For money back, the classifier matches Jev with about a thousand past complaints

Money-back complaints caught in the top tenth, percent, against past complaints used to train the classifier, weighted

Show data

Data for For money back, the classifier matches Jev with about a thousand past complaints
caught	conf_high	conf_low	labels
34	51.90	11.20	100 complaints
49.30	59.50	30.50	300 complaints
58.80	65.30	53.10	1,000 complaints
66.40	69.10	63.60	3,000 complaints
69.90	72.90	66.60	10,000 complaints
74.90	75.90	74.20	30,000 complaints

Read down the rows: each dot is the mean of refits on that many past complaints, and its whisker runs from the lowest to the highest refit, which is not a confidence interval. Each count includes the fifth held out for calibration. The vertical line is Jev, which uses none of them.Source: CFPB Consumer Complaint Database, 2025 distinct narratives with narratives of at most 3,000 characters; 10,000 sampled, weighted; engine calls October 2026; ten refits per size, three at the largest

The classifier catches 49% with three hundred past complaints, 59% with a thousand and 66% with three thousand. That is far fewer labels than the review task needed, and a desk already has these records, because its own case system stores how each complaint closed.

These counts include calibration labels held out from each sample, and the complaints being scored are not used. They describe this classifier on these records, so treat them as a guide for a pilot. Whether a desk should build this model also depends on whether outcomes arrive in time, whether old records resemble today's complaints, and whether money back is what the desk wants to prioritize.

What these data cannot show

The routing score compares the engine with the consumer's own product choice, so some of what looks like engine error is consumer error, and these data cannot say how much. The money-back label is what the company did, which depends on the company as much as the complaint. The costs are assumptions, so the comparisons show how policies trade off under them and are not measured savings. The sample describes distinct narratives, not every filing, and money back is more common among them than among all filed complaints. The complaints are limited to narratives of at most 3,000 characters, which leaves out 8% of 2025 complaints with a narrative, because the public narratives were withdrawn in August 2026 and only a cut copy remains.

Data sources

CFPB Consumer Complaint Database10,000 complaints received in 2025 with a narrative of at most 3,000 characters, stratified by product; complaints from August 2023 to 2024 for the classifierWeights: sample_weight restores the product mix of the 2025 complaints within the length limit · License: Public domain; the booklet shows aggregates only

All data sources

Methods

Two decision engines and a general language model read each sampled complaint narrative and answered two typed questions: which product it is about, and whether the consumer describes money the company could return. A classifier trained on earlier complaints answered the same two. Nothing was rated by hand.

Which complaints

CFPB withdrew complaint narratives from public access in August 2026, so the only copy available keeps the first 3,000 characters of each narrative. The sample keeps only complaints whose whole narrative fits, which excludes 7.7% of 2025 complaints with a narrative. Every result describes complaints within that length.

The sample holds the same number of complaints for each of the eleven products, and weights restore each product's share of 2025 complaints.

Labels

Routing is scored against the product the consumer selected when filing. Consumers sometimes pick the wrong product, so a disagreement can be the consumer's mistake rather than the engine's.

The money-back question is scored against the company's recorded response, closed with monetary relief. That outcome depends on the company as well as the complaint.

Intervals

Intervals are 95% bootstrap intervals that resample complaints within product strata, keep each product's share fixed and use the same resamples for every policy. They describe sampling error for 2025 complaints and say nothing about other years or other companies.

How to cite

@misc{singh2026resolvingandroutingcustomercomplaints,
  author = {Singh, Vishal},
  title = {Resolving and routing customer complaints},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/resolving-and-routing-customer-complaints},
  note = {In Decision AI. Published 29 October 2026.}
}