Chapter 1.1 · Report
Decision AI in applications and agents
An application can act on a model's judgment only through a stated policy, and in the opening bug-report example that policy routes a report to an engineer whenever the chance it can be reproduced exceeds 0.556.
Chapter page: the same findings with every figure drawn as a chart.
Every estimate of the chapter with its interval and every figure as a table, generated from the chapter's data files. Table n of this report is figure n of the chapter. The same report as Markdown: report.md.
Identification
- Title
- Decision AI in applications and agents
- Chapter
- 1.1 (Part I, From judgment to action; Case studies)
- Question
- Where do bounded judgments with a fixed answer space occur in applications and agents, and what does each kind of output establish?
- Dek
- An application can act on a model's judgment only through a stated policy, and in the opening bug-report example that policy routes a report to an engineer whenever the chance it can be reproduced exceeds 0.556.
- Grain
- records
- Status
- draft
- Published
- Updated
- not revised
- Data sources
- Authored examples. The bug-report example and its costs are written for this booklet; weights: not applicable; no data are sampled
- Canonical URL
- https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents
- Markdown
- https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents/report.md
- BibTeX
@misc{singh2026decisionaiinapplicationsandagents, author = {Singh, Vishal}, title = {Decision AI in applications and agents}, year = {2026}, url = {https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents}, note = {In Decision AI. Published 29 October 2026.} }
Principal estimates
The facts the dek and the opening prose cite, in order of appearance. Every fact is in the table of all estimates below.
| Estimand | Population | Period | Value | Interval | n | Weights | Source | Fact id |
|---|---|---|---|---|---|---|---|---|
| break-even probability, 1 minus delay cost over misrouting cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | 0.556 | not applicable; assumed inputs | Authored examples | break_even_p_repro | ||
| assumed cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | $90 | not applicable; assumed inputs | Authored examples | cost_misroute | ||
| assumed cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | $40 | not applicable; assumed inputs | Authored examples | cost_delay | ||
| assumed probability | one illustrative bug report (assumed) | assumed inputs, October 2026 | 0.6 | not applicable; assumed inputs | Authored examples | p_repro_example | ||
| (1 - p) times the misrouting cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | $36 | not applicable; assumed inputs | Authored examples | expected_cost_route |
Findings
A bug report reaches a software team, and someone has to decide whether to route it to an engineer now or ask the reporter for steps that reproduce the problem. Suppose that routing a report nobody can reproduce wastes $90$90 US dollars of engineer time per report routed that cannot be reproducedEstimandassumed costWeightsnot applicable; assumed inputsPopulationone illustrative bug report (assumed)Periodassumed inputs, October 2026SourceAuthored examplesNoteAssumed: one engineer hour. of an engineer's time before the engineer sends it back for details, and that asking first costs $40$40 US dollars per report sent back to the reporterEstimandassumed costWeightsnot applicable; assumed inputsPopulationone illustrative bug report (assumed)Periodassumed inputs, October 2026SourceAuthored examplesNoteAssumed: one day of delay, charged whether or not the report was reproducible. of delay whether or not the report was complete. A model reads the report and returns the probability that it can be reproduced.
Routing costs the misrouting cost only when the report cannot be reproduced, so a report with a probability of 0.60.6 probability that the report can be reproducedEstimandassumed probabilityWeightsnot applicable; assumed inputsPopulationone illustrative bug report (assumed)Periodassumed inputs, October 2026SourceAuthored examples has an expected costExpected lossThe average cost of an action, weighting each possible outcome's cost by its probability. of $36$36 expected US dollars per report routed nowEstimand(1 - p) times the misrouting costWeightsnot applicable; assumed inputsPopulationone illustrative bug report (assumed)Periodassumed inputs, October 2026SourceAuthored examples if routed, against $40$40 US dollars per report sent back to the reporterEstimandassumed costWeightsnot applicable; assumed inputsPopulationone illustrative bug report (assumed)Periodassumed inputs, October 2026SourceAuthored examplesNoteAssumed: one day of delay, charged whether or not the report was reproducible. if the team asks. Routing wins. The two actions cost the same at a probability of 0.5560.556 probability that the report can be reproducedEstimandbreak-even probability, 1 minus delay cost over misrouting costWeightsnot applicable; assumed inputsPopulationone illustrative bug report (assumed)Periodassumed inputs, October 2026SourceAuthored examples, so that is the cutoffCutoffThe probability above which a policy takes an action.: route above it and ask below it.
Drag the cutoff and watch the reports change lanes
- Misrouting costassumed
- $90
- Delay costassumed
- $40
- Break-even cutoffderived
- 0.556
- Routed now, of all reports
- 15 of 24
- Average expected cost at this cutoffderived
- $32.33
- At the break-even cutoffderived
- $31.57
With the cutoff at 0.40, 15 of 24 reports are routed now and 9 are sent back with a request. The average expected cost is $32.33 per report, against $31.57 at the break-even cutoff of 0.556, the lowest any cutoff reaches for these reports. At this cutoff, 4 reports sit on the costlier side of break-even.
The dots are illustrative probabilities from a seeded generator, not data. Both costs are assumptions. The arrow keys move the slider one step; dragging on the chart also moves the cutoff.
The example separates two things that are easy to run together. The probability is a judgment about the report, and it could come from a written rule, a trained classifier, a language model or a decision model. The cutoff is a policy, and it came from the costs. No model supplied it, and a better model would not change it. This booklet is about how judgments become actions, and about when a model's judgment improves the action compared with something simpler.
A decision system has more parts than its model
A model supplies one part of a decision system. The table follows the bug report through every part and names who owns each one.
| Part | In the bug-report example | Who owns it |
|---|---|---|
| Evidence | The report's text, the component it names, the reporter's earlier reports | The reporter and the issue tracker |
| Judgment | The probability that the report can be reproduced | A rule, a classifier or a model |
| Policy | Route when the probability exceeds the cutoff, otherwise ask | The engineering manager, who states the costs |
| Action | Route the report or send it back with a request | Application code |
| Outcome | Engineer time spent and days of delay | Recorded later in the tracker |
| Feedback | Whether a routed report was reproduced | Recorded only for reports that were routed |
One bug report through a decision system
Step 1 of 6: Evidence
The report's text, the component it names and the reporter's earlier reports.
The last row recurs in every case in this booklet. A team learns whether a routed report could be reproduced, because an engineer tried. It learns much less about the reports it sent back, because nobody tried them. The policy decides which outcomes the organization gets to observe, so the data available for judging the policy later are shaped by the policy itself.
Where the judgment can come from
Decision models put a bounded judgment behind an interface with a fixed answer space. The application supplies evidence and typed questions: a yes-or-no question, a choice among named options, or a score on a written rubric. The model returns a probability, a choice with a probability for each option, or a score, and ordinary code decides what to do with it. TypeSafe calls models of this kind "System One" models, borrowing Kahneman's term for fast, intuitive judgment [1]. Its Jev model is a prominent early example, and OpenAI offers a similar interface in its Decisions API [2]. Software libraries such as the AI SDK put several of these engines, and general language models, behind one function [3].Interfaces change faster than the ideas in this chapter, so the chapters name engines only where a result depends on one.
None of this makes the judgment new. Classification and statistical decision theory are much older, and a long line of research has found that simple statistical rules often predict as well as expert judgment does [4, 5]. A decision model is one more way to supply the judgment row of the table above. Each way produces an output that establishes something different.
| Output | What it establishes | What still needs checking |
|---|---|---|
| A decision model's answer | A choice, rubric score or probability from a fixed answer space | Whether it is correct and calibrated for this application's records |
| A language model's structured output | Text that fits a schema, within what the interface guarantees | Whether the content is correct, and whether any "confidence" field predicts anything |
| A tool call | A proposed operation and its arguments | Whether the operation is appropriate, permitted and likely to work |
| A rule or an executable test | A reproducible result for the condition it checks | Whether that condition captures the actual requirement |
Evidence that a policy works comes in kinds
A claim that a model "works" can mean any of several things, and each needs different evidence. The case chapters keep them apart.
| Kind of evidence | The question it answers | What can establish it |
|---|---|---|
| Judgment validity | Does the answer match a defensible reference label? | A labeled benchmark drawn from the records the policy will see |
| Probability validity | Among records given a probability near some value, does the event happen at about that rate? | The same benchmark, read for calibrationCalibrationWhether predicted probabilities match observed frequencies: among items given a probability of 0.3, about three in ten should turn out positive. in the relevant population |
| Policy value | Does acting on the answers lower expected cost under stated costs and constraints? | A comparison of complete policies on the benchmark, with the costs labeled as assumed |
| Observed effect | Did running the policy improve what actually happened? | Executing the policy and measuring outcomes, ideally against a randomized comparison |
A labeled benchmark can establish the first two kinds. A cost comparison on the benchmark shows the third, and only under its assumed costs. Neither shows the fourth. The cases climb this ladder in order. Chapter 1.2, From evidence to typed questions asks about judgment validity, Chapter 1.3, From probabilities to actions adds probability validity and policy value, Chapter 2.1, Resolving and routing customer complaints asks about policy value when a person's time is limited, and Chapter 3.1, Deciding when an agent is finished runs its policies and observes what happens, on tasks written for this booklet.
Four quantities that are easy to confuse
The probability that a label is true, a vendor's confidence statistic, the probability that an action succeeds, and a rubric score are different quantities. A decision model that says a complaint is about a mortgage with probability near one has said nothing about whether routing it to the mortgage team will resolve it. A confidence field may be computed from the spread of the answer probabilities and measure how peaked they are, which is a different quantity from the chance of being right. A rubric score places a record on a written scale, and no outcome has that score as its probability. TypeSafe's documentation makes a related point about its own outputs: calibration is a property of many predictions taken together, and it does not make any single answer correct [6].
Costs, constraints and the simplest alternative
Each case chapter reports the outcomes it measures first, such as missed escalations, hours of review or reports finished too early, and then shows how stated costs turn them into one comparison. The costs are assumptions, tagged as such, with a sentence on where a firm would find the real number. Some choices are not a matter of cost. An organization may require that certain complaints always reach a person, or that required tests always run. Those constraints rule out actions before any cost comparison starts, and a low assumed cost does not override them.
Every case also compares the model's policy with the simplest credible alternative: a keyword rule, a trained classifier, a fixed step limit, or sending everything to a person. A chapter can end with the simple policy matching or beating the model. That result is a finding about whether to buy or build the system, and the chapter reports it as one.
One report through a workflow
The bug report can continue past the first decision. Each step below poses a different kind of question, and each links the chapter that studies a decision of that kind.
| Step | Question at that step | Kind of answer | Studied in |
|---|---|---|---|
| Triage | Are the reproduction details sufficient? | Yes or no, with a probability | This chapter |
| Routing | Which team owns the report? | A choice among teams | Chapter 2.1, Resolving and routing customer complaints |
| Repair | What change fixes the problem? | Generated code | Outside this booklet's scope |
| Testing | Do the required tests pass? | An executable result | Outside this booklet's scope |
| Review | Does the change need a specialist's review? | Several yes-or-no questions, combined | Chapter 2.2, Allocating review attention to code changes |
| Stopping | Is the work finished, or should the agent continue or ask? | A choice among next actions | Chapter 3.1, Deciding when an agent is finished |
The same pattern appears outside software: a company decides which customer to contact after a bad review (Chapter 1.3, From probabilities to actions).
What these data cannot show
This chapter uses no data. Its example shows how a policy follows from a stated probability and stated costs, and says nothing about how often any engine returns a correct or calibrated probability. Whether a decision model's policy beats a rule or a classifier is a question each case chapter answers on its own records, and the answer can differ from case to case.
Tables
The chapter has no figures.
All estimates
Every fact in facts.json (5), sorted by id. The same values as JSON: facts.json.
| Fact id | Value | Unit | Estimand | Population | Period | Interval | n | Weights | Source | Note |
|---|---|---|---|---|---|---|---|---|---|---|
| break_even_p_repro | 0.556 | probability that the report can be reproduced | break-even probability, 1 minus delay cost over misrouting cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | not applicable; assumed inputs | Authored examples | |||
| cost_delay | $40 | US dollars per report sent back to the reporter | assumed cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | not applicable; assumed inputs | Authored examples | Assumed: one day of delay, charged whether or not the report was reproducible. | ||
| cost_misroute | $90 | US dollars of engineer time per report routed that cannot be reproduced | assumed cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | not applicable; assumed inputs | Authored examples | Assumed: one engineer hour. | ||
| expected_cost_route | $36 | expected US dollars per report routed now | (1 - p) times the misrouting cost | one illustrative bug report (assumed) | assumed inputs, October 2026 | not applicable; assumed inputs | Authored examples | |||
| p_repro_example | 0.6 | probability that the report can be reproduced | assumed probability | one illustrative bug report (assumed) | assumed inputs, October 2026 | not applicable; assumed inputs | Authored examples |
Definitions
- Expected loss
- The average cost of an action, weighting each possible outcome's cost by its probability.
- Cutoff
- The probability above which a policy takes an action.
- Calibration
- Whether predicted probabilities match observed frequencies: among items given a probability of 0.3, about three in ten should turn out positive.
Methods
No engine was called and no data were read for this chapter. The opening example states its costs and its probability as assumptions, and every other number on the page is arithmetic on those assumptions.
The bug-report example
Routing a report that cannot be reproduced is assumed to waste $90 of engineer time, after which the engineer sends it back for details; that second step is not counted. Asking the reporter first is assumed to cost $40 of delay whether or not the report could have been reproduced.
With p the probability that the report can be reproduced, routing costs (1 − p) times the misrouting cost in expectation and asking costs the delay cost. The two are equal at p = 1 − delay cost / misrouting cost, which is 0.556 at these inputs.
What these data cannot show
This chapter uses no data. Its example shows how a policy follows from a stated probability and stated costs, and says nothing about how often any engine returns a correct or calibrated probability. Whether a decision model's policy beats a rule or a classifier is a question each case chapter answers on its own records, and the answer can differ from case to case.
References
- TypeSafe (2026). System One. TypeSafe documentation, read October 24, 2026. docs.typesafe.ai/concepts/system-one
- OpenAI (2026). Decisions. OpenAI API documentation, read October 24, 2026. developers.openai.com/api/docs/guides/decisions
- Vercel (2026). Decisions. AI SDK Core documentation, read October 24, 2026. ai-sdk.dev/docs/ai-sdk-core/decisions
- Meehl, Paul E. (1954). Clinical versus Statistical Prediction. University of Minnesota Press.
- Dawes, Robyn M. (1979). The robust beauty of improper linear models in decision making. American Psychologist, 34(7), 571-582. doi:10.1037/0003-066X.34.7.571
- TypeSafe (2026). Confidence. TypeSafe documentation, read October 24, 2026. docs.typesafe.ai/confidence.md
Provenance
- Build script
- decision_ai/chapters/decision-ai-in-applications-and-agents/build.py (sha256 cd480dae075fe8e5385ca75321678ad77a4eb35316571f70d3301ad617c78ce2)
- Built on
- 2026-10-29
- Site commit
- 252942a900a49d01fe9f278e18459eb87b20b4a0
| File | SHA-256 |
|---|---|
| facts.json | 0aa468f3b6cc5d0c9f5a84a9622ac2d2d31ec50713c5c494e57d3b8091aea0b5 |
| meta.json | b233653547d7465ddda26fd43d738eb84873feb2090388e1764a897798ddf381 |
| refs.json | 508fd36e6b882c96fffea4f376a94a0b9e79ae78349bb37433a4427fac1baf5f |
Machine formats
- Report (Markdown): https://vishalsingh.org/decisionai/decision-ai-in-applications-and-agents/report.md
- Every estimate (JSON): https://vishalsingh.org/decisionai/report/decision-ai-in-applications-and-agents/facts.json
- Index of every report: https://vishalsingh.org/decisionai/llms.txt; all reports in one file: https://vishalsingh.org/decisionai/llms-full.txt