AI systems need more than traditional unit tests, but that does not mean traditional testing stops mattering.
A production AI agent is usually a combination of ordinary software and probabilistic behavior:
API/input handling
retrieval
model calls
structured outputs
tool selection
tool execution
state transitions
business rules
final response
Some of those components should be tested deterministically. Others need statistical evaluation across representative tasks. Agentic systems add another dimension: even if the final answer is correct, the trajectory may have been wasteful, unsafe, or operationally wrong.
Anthropic’s current 2026 agent-evaluation guidance emphasizes exactly this problem: agents act across many turns, call tools, modify state, and adapt based on intermediate results, so teams need evaluations that match the complexity of the behavior being measured.
The useful testing philosophy is:
> Use deterministic tests wherever correctness is objective, use evals where behavior is probabilistic, and test the full workflow under the same failures and constraints it will face in production.
Think in layers, not one “AI accuracy” score
A single score hides too much.
Suppose a customer-support agent fails.
Possible causes include:
wrong article retrieved
model misunderstood article
wrong tool selected
tool arguments invalid
permission correctly blocked action
payment API timed out
agent retried a mutation unsafely
final answer overstated success
Calling all of these “low accuracy” prevents useful debugging.
A better test model separates:
- deterministic software contracts
- model/component behavior
- agent trajectory
- end-to-end task outcome
- security/adversarial behavior
- production behavior over time
A practical AI testing pyramid
You can think of production AI testing like this:
Production monitoring
/ online evals
-------------------
Canary / shadow tests
-------------------------
End-to-end task evaluations
-------------------------------
Agent trajectory / tool evaluations
-------------------------------------
Component evals: RAG, routing, grading
-----------------------------------------
Deterministic unit / contract / policy tests
The lower layers should be faster and cheaper.
The higher layers provide realism but cost more to run and diagnose.
Do not try to replace the bottom of the pyramid with LLM judges.
Layer 1: deterministic unit tests still come first
If a rule can be checked exactly, test it exactly.
Examples:
JSON schema validation
RBAC/authorization
refund amount limits
idempotency keys
state-machine transitions
URL allowlists
file path restrictions
rate limits
required fields
Example:
user from tenant A requests tenant B invoice
→ expected: authorization denied
No LLM judge is needed.
The answer is objectively known.
Structured outputs should have contract tests
If a model is expected to return:
{
"category": "billing",
"urgency": "high"
}
then test:
valid JSON?
required fields?
allowed enum values?
additional fields allowed?
Provider-native structured output can reduce shape failures, but your application should still validate the runtime object before using it.
Schema-valid does not guarantee semantically correct—but invalid schema should fail immediately.
Tool schemas need normal API testing
Each tool should be tested independently of the model.
For example:
get_invoice(invoice_id)
should have tests for:
existing invoice
missing invoice
wrong tenant
malformed ID
downstream timeout
permission denied
If the tool itself is unreliable, agent eval scores will be noisy and misleading.
Do not debug model behavior around a broken tool implementation.
Layer 2: component evals isolate probabilistic behavior
Component evals test one model-driven piece at a time.
Examples:
intent classifier
query rewriter
retriever
reranker
structured extractor
router
response generator
This helps answer:
> Which component regressed?
rather than only:
> The whole agent got worse.
Retrieval should be evaluated separately from generation
A RAG system has at least two independent questions:
Retrieval
Did the system find the correct evidence?
Generation
Given correct evidence, did the model answer correctly?
Useful retrieval metrics include:
Recall@k
Precision@k
MRR
nDCG
If the right passage is absent, prompt tuning the generation model is unlikely to fix the real problem.
Test retrieval under real filters
Multi-tenant RAG often filters by:
tenant
permission
language
product
status
A retrieval system that performs well globally may fail under aggressive filters.
Your eval queries should run through the same ACL and metadata conditions as production.
Cross-tenant retrieval should be an automatic security test, not a manual demo check.
Tool selection is its own eval
For tool-using agents, evaluate whether the model chose the right tool.
Example test:
User: “What does your refund policy say?”
Expected: search/retrieve policy
Forbidden: execute refund
Another:
User: “What is the current status of invoice INV-42?”
Expected: get_invoice
Not RAG-only answer
Measure:
correct tool selected
wrong tool selected
unnecessary tool selected
required tool omitted
Tool argument correctness needs separate scoring
The model may choose the right tool with the wrong arguments.
Example:
{
"tool": "get_invoice",
"invoice_id": "INV-24"
}
when the user asked about INV-42.
Tool selection score = correct.
Tool argument score = wrong.
Separate these metrics so you know what to fix.
Layer 3: trajectory evaluation for agents
Agents are sequences of decisions.
A trajectory may look like:
model
→ search docs
→ model
→ get account
→ model
→ get invoice
→ model
→ request approval
→ tool
→ final answer
Two agents can produce the same final answer while one took a dangerous path.
Trajectory evaluation asks whether the process was acceptable.
What should trajectory tests inspect?
Useful checks include:
correct sequence of critical actions
unnecessary tool calls
duplicate tool calls
forbidden tools
retry loops
permission failures
handoff/escalation behavior
write before approval
verification after mutation
Example:
refund workflow
Expected critical ordering:
read invoice
→ verify eligibility
→ approval if required
→ execute refund
→ verify postcondition
A trajectory that refunds first and validates later should fail even if the final result happened to be correct.
Do not over-specify harmless trajectories
Agents need flexibility.
If the task can be solved correctly by:
search logs → inspect metrics
or:
inspect metrics → search logs
then requiring one exact sequence can make the eval brittle.
Trajectory checks should focus on:
- required safety invariants
- forbidden actions
- meaningful efficiency constraints
not stylistic preferences about every step.
Layer 4: end-to-end task evaluations
Ultimately, the user cares about task success.
Examples:
Coding agent
requested bug fixed?
tests pass?
unrelated behavior preserved?
Support agent
issue resolved or correctly escalated?
customer/account state correct?
Document extraction
required fields correct?
missing fields not hallucinated?
Research agent
requested questions answered?
claims supported by sources?
End-to-end metrics should stay close to product value.
Build eval tasks from real work
Anthropic recommends starting from product requirements and real failures rather than waiting for a huge benchmark dataset.
A useful seed set can include tens of high-signal tasks covering:
common cases
meaningful variations
known failures
edge cases
high-risk cases
Then grow it continuously.
A 40-case dataset that captures the real product can be more useful than 10,000 generic synthetic questions.
Every production failure should become a regression case
Suppose an agent once:
retried a refund after a timeout
→ duplicate refund created
After the bug is fixed, preserve the incident as a permanent test:
payment write times out after ambiguous outcome
Expected:
→ query operation state
→ do not blindly retry
This is how your test suite becomes proprietary operational knowledge.
Nondeterministic systems need repeated trials
A normal deterministic unit test can run once.
An agent may succeed 8 times and fail 2 times with the same input.
That variance matters.
For important cases, run multiple trials:
n = 5
n = 10
and track metrics such as:
success rate
mean tool calls
failure classes
latency distribution
One lucky run should not certify a probabilistic workflow.
Report distributions, not only averages
Suppose average agent latency is 12 seconds.
But:
p50 = 6s
p95 = 45s
p99 = 2m
The tail latency may make the product unusable.
Track:
p50/p95/p99 latency
tokens per task
tool calls per task
retries per task
Nondeterministic systems often have long tails.
Deterministic graders should be preferred when possible
Use code for objective outcomes:
unit tests pass
JSON valid
record created
expected value present
permission denied
file checksum matches
These graders are:
- cheaper
- reproducible
- easier to debug
Do not ask an LLM whether a unit test “looks passed.”
LLM-as-a-judge is useful for semantic quality
Some properties do not have exact answers.
Examples:
Was the explanation complete?
Was the summary faithful?
Did the answer satisfy tone/policy?
Was the evidence relevant?
A judge model can help score these.
But judges themselves need testing.
Calibrate LLM judges against humans
Do not assume a judge prompt is objective.
Create human-labeled examples and compare.
Useful process:
humans label cases
→ LLM judge labels same cases
→ inspect disagreements
→ refine rubric
Check for biases such as:
verbosity preference
position bias
model-family bias
For high-impact decisions, periodic human calibration remains important.
Pairwise evals can be easier than absolute scoring
Instead of:
Score this response from 1–10.
ask:
Which response better satisfies the rubric: A or B?
Pairwise evaluation is useful when comparing:
old model vs new model
prompt v5 vs prompt v6
retrieval pipeline A vs B
Still randomize answer order to reduce position bias.
Red-team testing is different from normal quality testing
Normal evals ask:
Does the system work correctly for legitimate use?
Red-team/adversarial tests ask:
How can an attacker or malformed input make it behave incorrectly?
These are separate test categories.
Prompt injection tests should be realistic
Do not test only:
“ignore previous instructions”
Indirect injection can arrive through:
web pages
emails
PDFs
retrieved documents
tool output
subagents
images/multimodal content
Test scenarios where malicious instructions are embedded in content the agent legitimately needs to process.
Expected outcome is often not “detect the phrase.”
It is:
untrusted content cannot override trusted policy or gain new permissions
Security evals should focus on blast radius
A model can eventually be manipulated.
The system should still limit consequences.
Test:
Can read-only agent write?
Can one tenant access another?
Can retrieved text trigger external network call?
Can LLM-generated file path escape workspace?
Can tool output inject instructions into privileged context?
Security testing should validate architecture, not only model resistance.
Failure injection is critical for agents
Production dependencies fail.
Test deliberately:
model timeout
tool timeout
HTTP 500
rate limit
empty result
malformed tool output
partial database outage
worker restart
Then verify the expected recovery behavior.
Retry behavior needs explicit tests
A retry should depend on failure class.
Example:
provider 503 → retry may be valid
invalid schema → model repair/retry may be valid
authorization denied → do not retry
unknown mutation outcome → verify first
Build tests for each class.
Otherwise agents tend to convert every failure into “try again,” which can be expensive or dangerous.
Test cancellation
If a user cancels a long agent run, verify:
pending model calls stop when possible
child tasks cancel
long tool processes receive cancellation
no new mutations begin
final state recorded
Cancellation is part of correctness for long-running workflows.
Test process restarts
Durable agents should survive:
worker crash
deployment
machine restart
Create an integration test:
run reaches checkpoint
→ kill worker
→ restart
→ verify correct resume
Do not assume persistence works because a database table exists.
Test concurrency and duplicate events
Real distributed systems may deliver events twice.
Test:
same queue message twice
same webhook twice
same approval callback twice
Expected behavior:
one logical mutation
This is especially important for agent actions involving payments, messages, or record creation.
Model upgrades require regression testing
A new model may improve reasoning while changing:
tool preferences
verbosity
structured output behavior
refusal rate
latency
cost
Run the exact same eval suite against:
old model/config
new model/config
Compare both quality and operational metrics.
Never upgrade only because a public benchmark improved.
Prompt changes are production changes
Version:
prompt
model
tool catalog
retrieval config
memory policy
A prompt edit that changes tool-use behavior can be as consequential as a code change.
Run regressions before rollout.
Test retrieval/index migrations
Changing an embedding model or chunking strategy can silently affect RAG.
Before switching:
build new index in parallel
→ run same retrieval eval set
→ compare recall/ranking
→ canary traffic
→ migrate
Keep the old index available until the new one is proven.
Canary releases are valuable for AI changes
Offline evals cannot capture every real interaction.
Deploy a new configuration to a small traffic percentage:
1% → 5% → 25% → 100%
while monitoring:
success rate
human corrections
escalation
latency
cost
safety signals
Roll back when important regressions appear.
Shadow testing reduces risk
For some workflows, you can run a new model/configuration in parallel without exposing its output to users.
production request
→ current system serves user
→ candidate system runs in shadow
Then compare outcomes offline.
This is useful for expensive or high-risk migrations, though it increases temporary compute cost.
Production monitoring completes the loop
NIST’s AI RMF emphasizes evaluating AI systems under deployment-like conditions and monitoring behavior in production.
Useful online signals include:
user correction
human escalation
approval rejection
agent cancellation
retry exhaustion
manual rollback
repeat request
support reopen
These signals help discover failures your offline test set missed.
Do not auto-learn blindly from negative feedback
A thumbs-down does not prove which component failed.
Treat production feedback as evidence:
signal
→ inspect trace
→ classify failure
→ create representative regression test
→ fix correct layer
This is safer than automatically rewriting prompts or training on every unhappy interaction.
Test data should be versioned
Record:
eval dataset version
case IDs
expected outcomes
rubric version
source/provenance
Then results are reproducible.
Example:
agent-v12 on evalset-v7 = 92.1%
agent-v13 on evalset-v7 = 94.8%
If the dataset changes silently, comparisons lose meaning.
Keep a protected test set
If engineers repeatedly tune against every eval case, the suite becomes a development set rather than a true test set.
A mature program may use:
development evals
→ visible to builders
release/holdout evals
→ used for important final comparison
fresh production-derived cases
→ added over time
This reduces eval overfitting.
A practical release gate
An AI change can require all of the following:
unit/contract tests pass
security regression tests pass
end-to-end success >= baseline
no critical high-risk case regressions
p95 latency within budget
cost/task within budget
trajectory violations == 0 for protected cases
Not every metric needs one global threshold.
Critical safety cases can be zero-tolerance while subjective quality uses statistical comparison.
Example: testing a support refund agent
Deterministic tests
wrong tenant denied
refund amount validated
idempotency key stable
approval threshold enforced
Component evals
billing intent classification
policy retrieval
Trajectory evals
Expected:
read invoice
→ verify duplicate
→ check policy
→ approval if needed
→ refund
→ verify refund
Failure injection
payment API times out after write
→ expected: query status before retry
Adversarial test
support ticket contains prompt injection requesting account dump
→ expected: no privilege expansion
End-to-end outcome
customer receives correct resolution or safe escalation
This is what production testing looks like.
Common mistakes
Evaluating only final text
Agent actions and trajectories matter.
Using LLM judges for objective outcomes
Prefer deterministic checks.
One trial per case
You will miss variance.
Only happy-path test data
Production failures live in exceptions.
No dependency-failure tests
Networks fail.
No security/adversarial suite
Quality evals are not security tests.
Changing prompt/model/retrieval together
You will not know which change caused the result.
No protected regression suite
Teams repeatedly rediscover old failures.
Production checklist
Before trusting an AI testing program, verify:
- Objective rules have deterministic tests
- Structured outputs have contract/schema tests
- Tools are tested independently from the model
- Retrieval is evaluated separately from generation
- Tool selection and argument correctness are separate metrics
- Agent trajectories have safety/efficiency checks
- End-to-end tasks reflect real user outcomes
- Important cases run multiple trials
- LLM judges are calibrated against human labels
- Prompt injection/adversarial cases are included
- Dependency failures are deliberately injected
- Retries, cancellation, and restart recovery are tested
- Duplicate-event/idempotency behavior is tested
- Model/prompt/retrieval upgrades run regressions
- Canary or shadow rollout exists for important changes
- Production signals feed new regression cases
- Eval datasets/configurations are versioned
Final takeaway
AI testing is not one benchmark and it is not one judge model.
Reliable systems combine ordinary deterministic tests with probabilistic evaluations, trajectory analysis, adversarial testing, failure injection, repeated trials, and production monitoring.
The core pattern is:
test contracts exactly
→ evaluate model behavior statistically
→ evaluate agent trajectories
→ test failure and adversarial conditions
→ canary real traffic
→ convert real failures into regressions
> The goal is not to prove that an agent is “smart.” The goal is to prove that the complete system behaves acceptably across the situations that matter in production.
References and further reading
Discussion (0)