Chapter 1.1

Decision AI in applications and agents

An application can act on a model's judgment only through a stated policy, and in the opening bug-report example that policy routes a report to an engineer whenever the chance it can be reproduced exceeds 0.556.

Author: Vishal Singh
Affiliation: NYU Stern
Published: 29 October 2026

Full report: every estimate with its interval, and every figure as a table.

DraftFigures and numbers may change as the analysis is rebuilt.what is still being checked

First draft. This chapter is conceptual: its numbers are assumed inputs and arithmetic on them. Worked examples from the complaint, code-review and agent chapters are added when those chapters have results.

A bug report reaches a software team, and someone has to decide whether to route it to an engineer now or ask the reporter for steps that reproduce the problem. Suppose that routing a report nobody can reproduce wastes $90 of an engineer's time before the engineer sends it back for details, and that asking first costs $40 of delay whether or not the report was complete. A model reads the report and returns the probability that it can be reproduced.

Routing costs the misrouting cost only when the report cannot be reproduced, so a report with a probability of 0.6 has an expected cost of $36 if routed, against $40 if the team asks. Routing wins. The two actions cost the same at a probability of 0.556, so that is the cutoff: route above it and ask below it.

Illustrationsynthetic

Drag the cutoff and watch the reports change lanes

Misrouting costassumed: $90
Delay costassumed: $40
Break-even cutoffderived: 0.556

Cutoff: route above it, ask below it

Routed now, of all reports: 15 of 24
Average expected cost at this cutoffderived: $32.33
At the break-even cutoffderived: $31.57

With the cutoff at 0.40, 15 of 24 reports are routed now and 9 are sent back with a request. The average expected cost is $32.33 per report, against $31.57 at the break-even cutoff of 0.556, the lowest any cutoff reaches for these reports. At this cutoff, 4 reports sit on the costlier side of break-even.

The dots are illustrative probabilities from a seeded generator, not data. Both costs are assumptions. The arrow keys move the slider one step; dragging on the chart also moves the cutoff.

The example separates two things that are easy to run together. The probability is a judgment about the report, and it could come from a written rule, a trained classifier, a language model or a decision model. The cutoff is a policy, and it came from the costs. No model supplied it, and a better model would not change it. This booklet is about how judgments become actions, and about when a model's judgment improves the action compared with something simpler.

A decision system has more parts than its model

A model supplies one part of a decision system. The table follows the bug report through every part and names who owns each one.

Part	In the bug-report example	Who owns it
Evidence	The report's text, the component it names, the reporter's earlier reports	The reporter and the issue tracker
Judgment	The probability that the report can be reproduced	A rule, a classifier or a model
Policy	Route when the probability exceeds the cutoff, otherwise ask	The engineering manager, who states the costs
Action	Route the report or send it back with a request	Application code
Outcome	Engineer time spent and days of delay	Recorded later in the tracker
Feedback	Whether a routed report was reproduced	Recorded only for reports that were routed

Illustration

One bug report through a decision system

Follow a report that isRouted nowSent back with a request

Step 1 of 6: Evidence

The report's text, the component it names and the reporter's earlier reports.

The last row recurs in every case in this booklet. A team learns whether a routed report could be reproduced, because an engineer tried. It learns much less about the reports it sent back, because nobody tried them. The policy decides which outcomes the organization gets to observe, so the data available for judging the policy later are shaped by the policy itself.

Where the judgment can come from

Decision models put a bounded judgment behind an interface with a fixed answer space. The application supplies evidence and typed questions: a yes-or-no question, a choice among named options, or a score on a written rubric. The model returns a probability, a choice with a probability for each option, or a score, and ordinary code decides what to do with it. TypeSafe calls models of this kind "System One" models, borrowing Kahneman's term for fast, intuitive judgment [1]. Its Jev model is a prominent early example, and OpenAI offers a similar interface in its Decisions API [2]. Software libraries such as the AI SDK put several of these engines, and general language models, behind one function [3].Interfaces change faster than the ideas in this chapter, so the chapters name engines only where a result depends on one.

None of this makes the judgment new. Classification and statistical decision theory are much older, and a long line of research has found that simple statistical rules often predict as well as expert judgment does [4, 5]. A decision model is one more way to supply the judgment row of the table above. Each way produces an output that establishes something different.

Output	What it establishes	What still needs checking
A decision model's answer	A choice, rubric score or probability from a fixed answer space	Whether it is correct and calibrated for this application's records
A language model's structured output	Text that fits a schema, within what the interface guarantees	Whether the content is correct, and whether any "confidence" field predicts anything
A tool call	A proposed operation and its arguments	Whether the operation is appropriate, permitted and likely to work
A rule or an executable test	A reproducible result for the condition it checks	Whether that condition captures the actual requirement

Evidence that a policy works comes in kinds

A claim that a model "works" can mean any of several things, and each needs different evidence. The case chapters keep them apart.

Kind of evidence	The question it answers	What can establish it
Judgment validity	Does the answer match a defensible reference label?	A labeled benchmark drawn from the records the policy will see
Probability validity	Among records given a probability near some value, does the event happen at about that rate?	The same benchmark, read for calibration in the relevant population
Policy value	Does acting on the answers lower expected cost under stated costs and constraints?	A comparison of complete policies on the benchmark, with the costs labeled as assumed
Observed effect	Did running the policy improve what actually happened?	Executing the policy and measuring outcomes, ideally against a randomized comparison

A labeled benchmark can establish the first two kinds. A cost comparison on the benchmark shows the third, and only under its assumed costs. Neither shows the fourth. The cases climb this ladder in order. Chapter 1.2, From evidence to typed questions asks about judgment validity, Chapter 1.3, From probabilities to actions adds probability validity and policy value, Chapter 2.1, Resolving and routing customer complaints asks about policy value when a person's time is limited, and Chapter 3.1, Deciding when an agent is finished runs its policies and observes what happens, on tasks written for this booklet.

Four quantities that are easy to confuse

The probability that a label is true, a vendor's confidence statistic, the probability that an action succeeds, and a rubric score are different quantities. A decision model that says a complaint is about a mortgage with probability near one has said nothing about whether routing it to the mortgage team will resolve it. A confidence field may be computed from the spread of the answer probabilities and measure how peaked they are, which is a different quantity from the chance of being right. A rubric score places a record on a written scale, and no outcome has that score as its probability. TypeSafe's documentation makes a related point about its own outputs: calibration is a property of many predictions taken together, and it does not make any single answer correct [6].

Costs, constraints and the simplest alternative

Each case chapter reports the outcomes it measures first, such as missed escalations, hours of review or reports finished too early, and then shows how stated costs turn them into one comparison. The costs are assumptions, tagged as such, with a sentence on where a firm would find the real number. Some choices are not a matter of cost. An organization may require that certain complaints always reach a person, or that required tests always run. Those constraints rule out actions before any cost comparison starts, and a low assumed cost does not override them.

Every case also compares the model's policy with the simplest credible alternative: a keyword rule, a trained classifier, a fixed step limit, or sending everything to a person. A chapter can end with the simple policy matching or beating the model. That result is a finding about whether to buy or build the system, and the chapter reports it as one.

One report through a workflow

The bug report can continue past the first decision. Each step below poses a different kind of question, and each links the chapter that studies a decision of that kind.

Step	Question at that step	Kind of answer	Studied in
Triage	Are the reproduction details sufficient?	Yes or no, with a probability	This chapter
Routing	Which team owns the report?	A choice among teams	Chapter 2.1, Resolving and routing customer complaints
Repair	What change fixes the problem?	Generated code	Outside this booklet's scope
Testing	Do the required tests pass?	An executable result	Outside this booklet's scope
Review	Does the change need a specialist's review?	Several yes-or-no questions, combined	Chapter 2.2, Allocating review attention to code changes
Stopping	Is the work finished, or should the agent continue or ask?	A choice among next actions	Chapter 3.1, Deciding when an agent is finished

The same pattern appears outside software: a company decides which customer to contact after a bad review (Chapter 1.3, From probabilities to actions).

What these data cannot show

This chapter uses no data. Its example shows how a policy follows from a stated probability and stated costs, and says nothing about how often any engine returns a correct or calibrated probability. Whether a decision model's policy beats a rule or a classifier is a question each case chapter answers on its own records, and the answer can differ from case to case.

Data sources

Authored examplesThe bug-report example and its costs are written for this bookletWeights: not applicable; no data are sampled · License: Authored for this booklet

All data sources

Methods

No engine was called and no data were read for this chapter. The opening example states its costs and its probability as assumptions, and every other number on the page is arithmetic on those assumptions.

The bug-report example

Routing a report that cannot be reproduced is assumed to waste $90 of engineer time, after which the engineer sends it back for details; that second step is not counted. Asking the reporter first is assumed to cost $40 of delay whether or not the report could have been reproduced.

With p the probability that the report can be reproduced, routing costs (1 − p) times the misrouting cost in expectation and asking costs the delay cost. The two are equal at p = 1 − delay cost / misrouting cost, which is 0.556 at these inputs.

References

TypeSafe (2026). System One. TypeSafe documentation, read October 24, 2026. docs.typesafe.ai/concepts/system-one
OpenAI (2026). Decisions. OpenAI API documentation, read October 24, 2026. developers.openai.com/api/docs/guides/decisions
Vercel (2026). Decisions. AI SDK Core documentation, read October 24, 2026. ai-sdk.dev/docs/ai-sdk-core/decisions
Meehl, Paul E. (1954). Clinical versus Statistical Prediction. University of Minnesota Press.
Dawes, Robyn M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34(7), 571-582. doi:10.1037/0003-066X.34.7.571
TypeSafe (2026). Confidence. TypeSafe documentation, read October 24, 2026. docs.typesafe.ai/confidence.md

How to cite

@misc{singh2026decisionaiinapplicationsandagents,
  author = {Singh, Vishal},
  title = {Decision AI in applications and agents},
  year = {2026},
  url = {https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents},
  note = {In Decision AI. Published 29 October 2026.}
}