Evaluate the call, including the audio.

Step 1

Prepare a local script.

Use Node 22 or later and a LiveKit server. Set LIVEKIT_URL, LIVEKIT_API_KEY, LIVEKIT_API_SECRET, and OPENAI_API_KEY in your local environment. Both sides of the call use a live model, so running the evaluation incurs provider usage.

npm install @animahealth/adk zod@3 @livekit/agents @livekit/agents-plugin-openai @livekit/rtc-node livekit-server-sdk
npm install --save-dev tsx

Create booking-eval.mts and add the following blocks in order. The voice setup guide explains the runtime dependencies.

Step 2

Define the agent and its simulated caller.

This lookup returns synthetic booking data. The same tool could serve a text agent; the realtime model configuration makes this agent speak. Its output tool ends the call.

import { z } from 'zod'
import { adk } from '@animahealth/adk'
import { openai } from '@animahealth/adk/openai'
import { voiceTimingMetric } from '@animahealth/adk/eval'

const app = adk()
const lookupBooking = app.tool({
	name: 'lookup_booking',
	description: 'Look up a booking by reference.',
	schema: z.object({ reference: z.string() }),
	execute: (ctx) => ({ reference: ctx.args.reference, slot: 'Tuesday 14:30' }),
})
const endCall = app.tool({
	name: 'end_call',
	description: 'End the call after answering the booking question.',
	schema: z.object({ summary: z.string() }),
	execute: (ctx) => ctx.end(),
})

const agent = app.agent({
	name: 'booking-support',
	model: openai.realtime('gpt-realtime-1.5', { voice: 'ballad' }),
	context: [
		app.context.system('Ask for the booking reference, look it up, and tell the caller the slot. Use end_call when finished.'),
		app.context.history(),
	],
	tools: [lookupBooking],
	output: endCall,
})
const caller = app.agent({
	name: 'booking-caller',
	model: openai.realtime('gpt-realtime-1.5', { voice: 'shimmer' }),
	context: [app.context.system('You are calling to check booking B-42. Ask what time it is. When answered, say thank you and goodbye.')],
	tools: [],
})

Step 3

Check the lookup and the response time.

Supply an explicit tool mock so the evaluation controls the lookup result. Let the real output tool end the call. The first metric checks that the agent looked up the requested reference and received the expected result. The second checks the 95th percentile of response latency against an example three-second budget. Choose a budget for your application.

const bookingCase = app.evaluate.voice.case({
	name: 'find-booking-time',
	agent,
	userAgent: caller,
	timeout: 120_000,
	toolMocks: {
		lookup_booking: {
			execute: (args) => ({ reference: args.reference, slot: 'Tuesday 14:30' }),
		},
		end_call: endCall,
	},
	metrics: [
		{
			name: 'looked-up-requested-booking',
			evaluate: (run) => ({
				passed: run.events.some((event) =>
					event.type === 'tool_call' && event.name === 'lookup_booking' &&
					event.args.reference === 'B-42' &&
					run.events.some((result) => result.type === 'tool_result' &&
						result.callId === event.callId && !result.error &&
						JSON.stringify(result.result) === JSON.stringify({ reference: 'B-42', slot: 'Tuesday 14:30' })),
				),
			}),
		},
		voiceTimingMetric({
			name: 'response-within-budget',
			measure: 'response_latency_p95',
			assertion: (ms) => ms < 3000,
		}),
	],
})

The lookup metric verifies tool behavior. It does not establish that the caller heard a correct answer. Review the recording and transcript, and add an outcome metric for the conversational requirement you need to enforce. A timing metric fails when its required measurements are missing.

Step 4

Run the call and review the evidence.

const suite = await app.evaluate.voice([bookingCase], {
	concurrency: 1,
	output: './booking-eval-results',
	hooks: [{
		onEnter: async (ctx) => {
			const reply = await ctx.voice.generateReply({ toolChoice: 'none' })
			await reply.waitForPlayout()
		},
	}],
})

console.log(app.evaluate.voice.report()(suite))
process.exitCode = suite.summary.passed === suite.summary.total ? 0 : 1

npx tsx booking-eval.mts

The evaluation creates a LiveKit room and connects the agent and simulated caller. The console report lists case and metric results. Each case's output directory contains a report.md and recording.wav. The returned case result also includes session events, voice events, a transcript, timing, and usage when available.

Expect the wording and timing to vary between runs. Inspect a failed metric's evidence, then use the recording and events to determine whether the problem was the conversation, a tool, or the connection. A timeout or errored call is not a passing evaluation.

Broaden the cases

Measure the situations your callers encounter.

• Add callers who forget their reference, change their request, or interrupt the agent.
• Use repeat to run each case several times and inspect variation.
• Add time_to_first_speech, silence_gap_max, or interruption_count metrics where they answer a product question.
• Use app.evaluate.voice.case((control) => ...) with control.disconnectUser() to exercise caller disconnection.

Start with the same business outcomes you check in text evaluations, then add audio and lifecycle requirements. Use deterministic tests for tool and state behavior that does not require a live conversation.