Trust, Evaluation, and Governance of Data Agents

The transcript can lie

Evaluation is harder for agents than for models, and for a reason worth internalizing. As Anthropic puts it, a flight-booking agent might end its transcript with “Your flight has been booked” — but the only outcome that matters is whether a reservation actually exists in the environment's database.¹ The transcript is what the agent says; the outcome is what the agent did, and the two can diverge. So practitioners separate trajectory evals (were the tool choices and reasoning steps sound?) from outcome evals (did the world end up in the right state?), and grade them with calibrated LLM-as-judge rubrics checked against a human gold set — giving the judge permission to answer “Unknown” rather than guess.¹ Academic surveys confirm the gaps: evaluation of cost, safety, and robustness remains immature.²

You can't govern what you can't see

Because an agent's path is discovered at run time, you cannot reason about it from the code alone — you have to record what actually happened. Observability for agents has standardized fast around OpenTelemetry's GenAI conventions: every run becomes a span tree with a top-level invoke_agent and child chat and execute_tool spans, capturing the model, the token counts, and the tool calls.³ Platforms like LangSmith and the open-source Arize Phoenix render that tree so a human can see exactly which query the agent ran and where it went wrong.⁴⁵

The lethal trifecta

The security failure mode unique to agents is prompt injection: because an agent treats the data it reads as potentially containing instructions, a hostile string hidden in a document or a database field can hijack it. The security researcher Simon Willison names the dangerous condition the “lethal trifecta”: access to private data, exposure to untrusted content, and the ability to communicate externally. Any one is fine; all three together let an attacker exfiltrate data through the agent itself.⁶ Prompt injection is the number-one risk in the OWASP Top 10 for LLM Applications,⁷ and it is not theoretical: Microsoft's 2025 “EchoLeak” flaw (CVE-2025-32711, scored a critical 9.3) let a single crafted email exfiltrate data from Microsoft 365 Copilot with no user interaction.⁸⁹

Guardrails and the human gate

The practical controls are layered. Input guardrails reject disallowed requests before the agent runs; output guardrails validate or redact what the agent produces before it leaves the system; and the most important control of all is the human approval gate. OpenAI's agent tooling lets any tool be flagged so that, instead of executing, it pauses the run into a resumable state for a person to approve or reject.¹⁰ For any action that touches money, customers, or production data, that gate is the difference between a useful agent and an unbounded liability.

The compliance backdrop

None of this happens in a vacuum. Three frameworks now anchor enterprise AI governance: the U.S. NIST AI Risk Management Framework, whose Generative AI Profile enumerates a dozen risk areas and hundreds of suggested actions;¹² the European Union's AI Act, whose obligations for general-purpose models began applying in August 2025 with most high-risk rules following in 2026;¹¹ and ISO/IEC 42001, the first certifiable management-system standard for AI, against which an organization can be independently audited.¹³