Evals: measuring agents you did not script

Step 1

A case, and the tools it must not really call

The agent under test is ordinary: one tool that reads, one tool that spends money. Nothing about it knows it is being evaluated. Build it once — every cell below scores this agent. Only the last cell on the page calls a real model, so the key box can stay empty until then.

import { adk } from '@animahealth/adk'
import { openai } from '@animahealth/adk/openai'
import { z } from 'zod'

const app = adk()

const orderStatus = app.tool({
  name: 'order_status',
  description: 'Look up an order and how late it arrived',
  schema: z.object({ orderId: z.string() }),
  execute: (ctx) => ({ orderId: ctx.args.orderId, daysLate: ctx.args.orderId === 'A-1' ? 7 : 0 }),
})

// The dangerous one. Its execute is deliberately unreachable, so a run that reaches it is loud.
const issueRefund = app.tool({
  name: 'issue_refund',
  description: 'Refund an order. Moves real money.',
  schema: z.object({ orderId: z.string(), amount: z.number() }),
  execute: () => {
    throw new Error('the real refund gateway was called')
  },
})

const support = app.agent({
  name: 'support',
  model: openai('gpt-5.6-luna'),
  context: [
    app.context.system('Refund an order only if it arrived more than three days late. Check it first.'),
    app.context.history(),
  ],
  tools: [orderStatus, issueRefund],
})

support.name

Testing replaced the model and let your tools run. An eval moves the other seam too: toolMocks replaces a tool's execute, so a suite can exercise the refund path a thousand times without a thousand refunds. A mock is { execute } — but note the signature, which is not a tool's: execute(args, ctx) takes the arguments first, where a real tool takes one ctx and reads ctx.args. Passing the real tool as the value instead lets it through untouched, which is what you want for a lookup with no side effects.

One case is one run plus the mocks it is allowed. app.evaluate takes a single case or an array; either way it answers with a suite result.

import { MockAdapter, type MockAdapterConfig } from '@animahealth/adk/testing'

// A fresh adapter per suite. One MockAdapter holds one cursor, so reusing it would make the
// second press of Run disagree with the first.
const offline = (responses: MockAdapterConfig['responses']) =>
  adk({
    adapters: {
      openai: new MockAdapter({ responses, defaultResponse: { text: 'the script ran out' } }),
    },
  })

const refundScript = [
  { toolCalls: [{ name: 'order_status', args: { orderId: 'A-1' } }] },
  { toolCalls: [{ name: 'issue_refund', args: { orderId: 'A-1', amount: 40 } }] },
  { text: 'Order A-1 was seven days late, so I have refunded 40.' },
]

const mocks = {
  order_status: orderStatus, // real: it only reads
  issue_refund: {
    execute: (args: unknown) => ({
      refunded: true,
      reference: `mock-${(args as { orderId: string }).orderId}`,
    }),
  },
}

const first = await offline(refundScript).evaluate({
  name: 'refunds a late order',
  runnable: support,
  input: 'Order A-1 turned up a week late. I want my money back.',
  toolMocks: mocks,
})

const outcome = {
  summary: first.summary,
  status: first.results[0].status,
  turns: first.results[0].turns,
  tools: first.results[0].events.flatMap((event) =>
    event.type === 'tool_result' ? [{ name: event.name, result: event.result }] : [],
  ),
}

outcome

The refund reference is mock-A-1, so the mock ran and the gateway did not. Delete the issue_refund entry and press Run again: toolMocks is strict, so a called tool with no entry throws inside the tool. That is an ordinary tool-phase error, and the default handler records it rather than propagating it — it lands in the ledger as a tool_result carrying an error that names the tool and prints the mock you should have written. Nothing escapes evaluate: the run answers the customer anyway, and the case still comes back passed.

Step 2

The metric is the question

A metric is a name and a function over the finished run: { name, evaluate(run) }, returning passed and optionally score, evidence, and data. It reads the ledger — run.session.events and run.session.state — because that is where everything the agent did already is. app.evaluate.metric is an identity helper that types the callback; it changes nothing at runtime.

import type { Event } from '@animahealth/adk'

const toolsCalled = (events: readonly Event[]) =>
  events.flatMap((event) => (event.type === 'tool_call' ? [event.name] : []))

const refundIssued = app.evaluate.metric({
  name: 'refund_issued',
  evaluate: (run) => {
    const called = toolsCalled(run.session.events)
    const passed = called.includes('issue_refund')
    return { passed, score: passed ? 1 : 0, evidence: [`called: ${called.join(' → ') || 'nothing'}`] }
  },
})

const refundDeclined = app.evaluate.metric({
  name: 'refund_declined',
  evaluate: (run) => {
    const refunds = toolsCalled(run.session.events).filter((name) => name === 'issue_refund')
    return { passed: refunds.length === 0, evidence: [`issue_refund calls: ${refunds.length}`] }
  },
})

// The invariant both cases owe: never refund without looking the order up first.
const checkedFirst = app.evaluate.metric({
  name: 'checked_first',
  evaluate: (run) => {
    const called = toolsCalled(run.session.events)
    const refundAt = called.indexOf('issue_refund')
    const lookupAt = called.indexOf('order_status')
    return {
      passed: refundAt === -1 || (lookupAt !== -1 && lookupAt < refundAt),
      evidence: [`order: ${called.join(' → ') || 'no tools'}`],
    }
  },
})

const questions = [refundIssued, refundDeclined, checkedFirst].map((metric) => metric.name)

questions

A metric that throws does not crash the suite: the runner catches it and records passed: false with the error as its evidence. Scores between 0 and 1 are averaged into the report; anything outside that range is carried but not averaged.

Four metric builders ship for the shapes you write most often. They live on the @animahealth/adk/eval subpath, which this page's substrate does not serve, so this block is shown rather than run — the cells above are the runnable form of the same contract.

import {
  eventCountMetric,
  eventSequenceMetric,
  stateMetric,
  timingMetric,
} from '@animahealth/adk/eval'

// How many events of a type match a filter?
eventCountMetric({
  name: 'refunded_once',
  eventType: 'tool_call',
  filter: (event) => event.name === 'issue_refund',
  assertion: (count) => count === 1,
})

// Did these events happen, in this order? (Other events in between are allowed.)
eventSequenceMetric({
  name: 'lookup_then_refund',
  sequence: [
    { eventType: 'tool_call', filter: (event) => event.name === 'order_status' },
    { eventType: 'tool_call', filter: (event) => event.name === 'issue_refund' },
  ],
})

// What did a state key end up as? (Replayed from the run's state_change events.)
stateMetric({
  name: 'case_closed',
  scope: 'session',
  key: 'closed',
  assertion: (value) => value === true,
})

// Was it fast enough?
timingMetric({
  name: 'answered_quickly',
  measure: 'time_to_first_assistant',
  assertion: (ms) => ms < 4000,
})

timingMetric's measure is a closed set: total_duration, time_to_first_assistant, time_to_first_tool_call, model_latency_total, model_latency_average, tool_execution_total, tool_execution_average. A measure it cannot compute — no matching events, missing timestamps — fails with that as its evidence rather than passing vacuously. The same subpath also ships codingDeltaMetric, for scoring a coding agent on the diff it produced rather than on its summary of the diff.

Step 3

A suite, and where its verdict comes from

Metrics attach in two places. Suite-level metrics in the options apply to every case — the invariants. Case-level metrics are added on top — the thing that case is actually about. Here one case must refund and one must not, and both owe the ordering invariant.

const declineScript = [
  { toolCalls: [{ name: 'order_status', args: { orderId: 'B-2' } }] },
  { text: 'Order B-2 arrived on time, so there is nothing to refund.' },
]

const refundCases = app.evaluate.cases([
  {
    name: 'refunds a late order',
    runnable: support,
    input: 'Order A-1 turned up a week late. I want my money back.',
    toolMocks: mocks,
    metrics: [refundIssued],
  },
  {
    name: 'leaves an on-time order alone',
    runnable: support,
    input: 'Order B-2 arrived on Tuesday as promised. Refund me anyway?',
    toolMocks: mocks,
    metrics: [refundDeclined],
  },
])

const suite = await offline([...refundScript, ...declineScript]).evaluate(refundCases, {
  metrics: [checkedFirst],
  concurrency: 1,
})

const scored = {
  summary: suite.summary,
  cases: suite.results.map((result) => ({
    name: result.name,
    status: result.status,
    metrics: result.metrics,
  })),
}

scored

app.evaluate.cases exists so a case array written in its own module is checked against the app's state schema at the point you write it, not the point you run it.

concurrency: 1 is load-bearing here only: one scripted cursor serves both cases. Against a real model the default of 64 applies.

A case's status is the run's outcome first, metrics second. If the run ended terminated, error, or aborted, that is the status and the metrics are decoration. Otherwise it is passed when every metric passed and failed when one did not — so a run that ends parked on a yield still passes, if what you asked about is true.

Step 4

The report

app.evaluate.report(options?) returns a function from a suite result to markdown: pass rate, per-metric rates with a mean of any 0–1 scores, then a Cases section that lists only the failures and their evidence. It is a pure function of the result — no rerun, no second pass over the model.

const markdown = app.evaluate.report({
  title: 'Refund suite',
  footer: (result) => `${result.results.length} cases in ${result.durationMs}ms`,
})(suite)

markdown

sections and renderCase customise the output when the audience is a reviewer rather than CI.

Each case result carries more than the report prints. run is the whole RunResult and events is its session ledger, so anything the test kit's helpers read, a failing case can be debugged with. turns counts the user messages in the run — one for a single-shot case, more once a simulated user is answering.

summary counts each of those across the suite as total, passed, failed, errors, terminated, aborted, timedOut. Only passed is good news; a suite that is 40% errors is not 60% healthy.

Step 5

Simulating the other side

Every case so far was one message in. Real behaviour needs a conversation, and a conversation needs someone on the other end. app.simulate is the loop app.evaluate runs each case through: run the agent, and while it is parked, answer it and run it again. A userAgent answers a message yield; a toolAgents entry, keyed by tool name, answers a yielding tool's question.

const chatSupport = app.agent({
  name: 'chat_support',
  model: openai('gpt-5.6-luna'),
  context: [app.context.system('Ask one question at a time.'), app.context.history()],
  tools: [],
  yields: true,
})

const customer = app.agent({
  name: 'customer',
  model: openai('gpt-5.6-luna'),
  context: [app.context.system('You are a customer chasing a refund. Answer briefly.'), app.context.history()],
  tools: [],
})

// Two agents interleaving: route by name so neither can eat the other's script.
const conversationAdapter = new MockAdapter({ defaultResponse: { text: 'the script ran out' } })
conversationAdapter.addResponses('agent:chat_support', [
  { text: 'Sorry about that — what is the order number?' },
  { text: 'Thanks. Was it more than three days late?' },
  { text: 'Then I can refund A-1 for you now.' },
])
conversationAdapter.addResponses('agent:customer', [{ text: 'A-1.' }, { text: 'Seven days late.' }])

const conversation = await adk({ adapters: { openai: conversationAdapter } }).simulate(chatSupport, {
  input: 'I want a refund.',
  userAgent: customer,
  maxTurns: 2,
})

const transcript = conversation.session.events.flatMap((event) => {
  if (event.type === 'user') return [`customer: ${event.text}`]
  if (event.type === 'assistant') return [`support: ${event.text}`]
  return []
})

const simulated = {
  status: conversation.status,
  stoppedBecause: conversation.status === 'terminated' ? conversation.terminationReason : undefined,
  transcript,
}

simulated

The customer's own turns are missing from that ledger on purpose: the simulated user runs in its own session, and only the message it produced crosses into the agent's. That is the whole trick — the agent under test cannot tell a simulated user from a real one, because it sees the same rows either way.

An agent with yields: true never completes on its own, so the loop needs a stop condition. maxTurns caps the exchanges, maxDuration caps the clock, and stateMatches stops as soon as the session state says the thing you were waiting for happened — { session: { refunded: true } }, with $exists, $eq, and $ne available for a key you want to test rather than match. Whichever fires, the result comes back terminated with the reason attached, and a case in a suite reports that status verbatim.

Two customisations sit either side of the loop. transform.prepareInput rewrites what the simulated user or tool agent is asked — by default the agent's last reply for a message yield, and the tool call as JSON for a tool yield. transform.processOutput rewrites what comes back, which is how a scripted answer table replaces a model entirely. A tool yield with no matching toolAgents entry also throws by name — but unlike a missing toolMocks entry, nothing records it. The throw escapes the loop, so simulate rejects and a case in a suite comes back error.

Step 6

Against a real model

Nothing above needed a key, and nothing above measured a model — a scripted adapter can only confirm the plumbing. Point the same cases at a real model and the suite starts answering the question it exists for: does this model, on this prompt, do the right thing? The mocks stay, so the refund gateway is still never called. The model is the only thing that changes. Paste a key into the box at the top of the page and press Run.

const live = await app.evaluate(refundCases, {
  metrics: [checkedFirst],
  onCase: (result, done, total) => console.log(`${done}/${total} ${result.name}: ${result.status}`),
})

app.evaluate.report({ title: 'Refund suite · live' })(live)

Now the numbers mean something, including the money ones: each case result carries usage, and the report totals tokens and cost across the suite when the provider reported them. Edit the system prompt in step 1, press Run again, and watch the pass rate and the bill move together. Run it with repeat: 5 and the report groups the runs per case — a metric that passes four times out of five is the fact a single green run was hiding.

Step 7

In your repo

An eval suite is an async function call and a number, so it needs no runner of its own. This block imports node:process, which this page's substrate does not serve, so it is shown rather than run.

import process from 'node:process'

import { app } from './app'
import { refundCases } from './cases'
import { checkedFirst } from './metrics'

const result = await app.evaluate(refundCases, {
  metrics: [checkedFirst],
  repeat: 5,
  onCase: (caseResult, done, total) =>
    process.stdout.write(`\r${done}/${total} ${caseResult.name}`),
})

process.stdout.write(`\n${app.evaluate.report({ title: 'Refund suite' })(result)}\n`)

// The gate is yours: `evaluate` reports, it does not exit. A CI suite that never fails the
// build is a dashboard, not a gate.
const bad = result.summary.failed + result.summary.errors + result.summary.timedOut
process.exit(bad > 0 ? 1 : 0)

That last line is the one teams forget. app.evaluate returns a result and never throws on a failed metric — deliberately, because a local exploration run wants the report more than it wants an exception. Anything blocking a merge has to read summary and decide.

Two boundaries worth naming. Deterministic behaviour belongs in scripted tests, not here: an eval that a script could have answered is just a slow, expensive test. And spoken conversations have their own surface: app.evaluate.voice, with recordings, transcripts, and turn timings — see voice agents.

Where this goes next: guardrails and recovery for the failures a suite keeps finding, and streaming and cost for the usage numbers the live report totalled.