Voice agents: the same agent, on the phone

Step 1

What actually changes

app.handler carries turn, rest, agui and voice. The first three are the request shape of Serving it: a config in, a function you call per request out. voice is the odd one. There is no request. A call arrives at a LiveKit room, and what the handler hands back is a worker.

Here is a whole voice program. One tool, one agent, one handler — and the last two lines, which are the part that is not like anything else in the package.

import { fileURLToPath } from 'node:url'

import { z } from 'zod'

import { adk } from '@animahealth/adk'
import { openai } from '@animahealth/adk/openai'

const app = adk({ name: 'bookings' })

const lookupStay = app.tool({
  name: 'lookup_stay',
  description: 'Look up a booking by its reference',
  schema: z.object({ ref: z.string() }),
  execute: (ctx) => ({ ref: ctx.args.ref, nights: 3 }),
})

const concierge = app.agent({
  name: 'concierge',
  model: openai.realtime('gpt-realtime-1.5', { voice: 'ballad' }),
  context: [app.context.system('Answer booking questions.'), app.context.history()],
  tools: [lookupStay],
  timeouts: { inactivity: 15_000, expiry: 300_000 },
})

const handler = app.handler.voice({ agent: concierge })

export default handler
handler.start(fileURLToPath(import.meta.url))

That file is both the worker and the thing that launches it. start(entryFile) boots a LiveKit agent server and tells it which module to load per job. LiveKit spawns a subprocess per call and reads that module's default export, which must carry an entry function and either a prewarm function or none. The handle is exactly that shape, so export default handler is the whole wiring. Inside a spawned subprocess start returns immediately instead of booting a second server. That is why one file can be both.

Run it the way LiveKit runs a worker — the subcommand is read from process.argv:

npx tsx call.ts dev      # reload on change, for local iteration
npx tsx call.ts start    # production
npx tsx call.ts connect  # join one named room and exit

Two constructor-time checks are worth knowing before you deploy. app.handler.voice requires @livekit/agents to be installed and throws when it is not — the core package never imports it, and the import happens the moment you call voice(). And the runnable must be an agent carrying a realtime model config; a sequence, a step, or an agent on an ordinary chat model is rejected with a message naming openai.realtime() and gemini.realtime().

Step 2

What has to be running

The ADK does not carry the media stack. LiveKit does: a room server that the caller and your worker both connect to, and — for a real phone number — a SIP trunk in front of it. Self-host it or use LiveKit Cloud; either way the worker finds it through three environment variables, read by the LiveKit CLI at boot.

LIVEKIT_URL=wss://your-project.livekit.cloud
LIVEKIT_API_KEY=…
LIVEKIT_API_SECRET=…
OPENAI_API_KEY=…            # or GOOGLE_API_KEY for a Gemini realtime model

The peers are opt-in, and each one buys a specific feature. Install the first two (or the Google plugin instead) and you have a working call; the rest are for what you add later.

npm install @livekit/agents @livekit/agents-plugin-openai

The last two rows arrive on their own: @livekit/agents depends on livekit-server-sdk and declares @livekit/rtc-node as its own peer. They are listed because the ADK loads them lazily, by name. A broken install therefore shows up as a missing feature, not a missing module: egress recording that logs and returns, local capture that warns and skips.

Step 3

The realtime model

A realtime config is a wrapper, not a model. RealtimeModelConfig holds realtime: true and the inner provider model. Around it go the audio-side settings the inner model has no vocabulary for: which voice, how turns are detected, how speech is transcribed, how input noise is handled. Three shapes come out of that wrapper. Which one you get is decided entirely by whether you pass stt and tts.

import { gemini } from '@animahealth/adk/gemini'
import { openai } from '@animahealth/adk/openai'
import { realtime } from '@animahealth/adk/voice'

// Speech in, speech out, one model.
const speechToSpeech = openai.realtime('gpt-realtime-1.5', {
  voice: 'ballad',
  turnDetection: { type: 'semantic', silenceDurationMs: 500 },
  inputTranscription: { model: 'gpt-4o-mini-transcribe' },
  noiseReduction: { type: 'near_field' },
})

const geminiLive = gemini.realtime('gemini-live-2.5-flash-native-audio', { voice: 'Puck' })

// Half-cascade: the realtime model reasons in text, your TTS speaks.
const halfCascade = openai.realtime('gpt-realtime-1.5', { tts: elevenLabsTTS })

// Full pipeline: an ordinary chat model between an STT and a TTS.
const pipeline = realtime({ model: openai('gpt-5.6-luna'), stt: deepgramSTT, tts: elevenLabsTTS })

stt and tts are LiveKit plugin instances, typed unknown by the ADK so the core carries no dependency on them. Anything else the provider accepts goes through providerOptions, which is merged last and therefore wins over everything the ADK computed.

Voice mode speaks to two providers. A realtime config on any other provider is rejected when the call starts, with the provider named. Gemini's Vertex settings ride on the inner model — gemini.realtime(name, { vertex: { project, location } }) — and are translated into the plugin's own Vertex options.

One environment note. Voice resolves its OpenAI connection separately from text mode and understands one override: OPENAI_EU_API_KEY routes the realtime connection at the EU endpoint. Otherwise it passes nothing and the LiveKit plugin reads OPENAI_API_KEY itself. Azure endpoints configured for text mode are not carried into a call.

Step 4

Which session the call joins

A request carries its own sessionId. A call carries a room and a participant, so the handler asks you. setup runs once, after the worker has connected to the room and the caller has joined, and its return value decides which session row this call appends to. Omit it and the room's name is the session id.

const app = adk({
  name: 'bookings',
  schema: {
    session: { caller: z.string().default('unknown'), channel: z.string().default('voice') },
  },
})

const handler = app.handler.voice({
  agent: concierge,
  name: 'concierge-voice',
  setup: (participant) => ({
    sessionId: participant.attributes?.['booking_ref'] ?? `call-${Date.now()}`,
    state: { caller: participant.identity ?? 'unknown', channel: 'voice' },
  }),
  sound: { noiseCancellation: 'telephony' },
  timeouts: { inactivity: 15_000, expiry: 600_000 },
})

participant is deliberately thin — identity and attributes, the two fields a SIP trunk or a token can populate. That is enough to route a caller back to the thread they were on last week. Return that thread's id and the agent picks up a history nobody had to hand it — the ledger was already there.

The rest of the config is the deployment's business, once, at boot:

interface VoiceHandlerConfig {
  agent: Runnable                       // must be an agent on a realtime model
  schema?: StateSchema
  setup?: (participant: VoiceParticipant) => SessionSetup | Promise<SessionSetup>
  sound?: SoundConfig                   // noise cancellation, thinking sound
  recording?: RecordingConfig           // { dir?, egress? }
  callTermination?: false | CallTerminationConfig
  hooks?: VoiceHook[]
  errorHandlers?: ErrorHandler[]
  timeouts?: AgentTimeouts              // { inactivity?, expiry? }
  name?: string                         // the name registered with LiveKit
  worker?: Record<string, unknown>      // extra LiveKit ServerOptions
  prewarm?: (proc: unknown) => void | Promise<void>
}

interface SessionSetup {
  sessionId: string
  scopes?: Partial<Record<SharedScope, string>>
  state?: Record<string, unknown>
  initialState?: StateChanges
  recordingKey?: string                 // overrides the egress object key
  noiseCancellation?: NoiseCancellationType
}

Four of those are worth a sentence each. timeouts.inactivity is measured from the last thing that happened, not the last thing the caller said, and it re-arms after every silence. An agent's own timeouts override the handler's, so a transfer can change the clock. sound.noiseCancellation is 'telephony' or 'general', set per handler or per session. recording has two independent halves. dir writes raw tracks locally and mixes them into a WAV on hangup, which is the development one; egress starts a LiveKit room-composite egress straight to S3, which is the production one. And callTermination is on by default: when the call completes the ADK deletes the room, or removes the participant under strategy: 'removeParticipant'. Set it to false when something else owns the hangup.

prewarm runs once per worker subprocess before any call reaches it — the place for tracing setup or loading something heavy, so the first caller does not wait for it.

Step 5

Tools during a call

A tool is the same object in both modes. The voice bridge converts each one into a LiveKit tool carrying the same JSON schema. The executor around it then does what the text runner does: parse the args, run prepare, call execute, run finalize, write a tool_call and a tool_result. Errors go to your error handlers with the same retry, fallback and throw vocabulary. Arguments that fail the schema come back to the model as text rather than throwing, so a bad call is a turn the model can correct.

// Same tool object, either handler. `ctx.voice` is the only voice-only field, and it is
// `undefined` in text mode.
const lookupStay = app.tool({
  name: 'lookup_stay',
  description: 'Look up a booking by its reference',
  schema: z.object({ ref: z.string() }),
  execute: async (ctx) => {
    await ctx.voice?.say('Let me pull that up.')
    return { ref: ctx.args.ref, nights: 3 }
  },
})

// Return a Runnable and the call transfers: LiveKit swaps the agent in place, mid-call.
const toBilling = app.tool({
  name: 'transfer_to_billing',
  description: 'Hand the caller to the billing agent',
  schema: z.object({}),
  execute: () => billing,
})

// The output tool. Calling it ends the call and becomes the run's output.
const endCall = app.tool({
  name: 'end_call',
  description: 'End the call once the caller is done',
  schema: z.object({ summary: z.string(), resolved: z.boolean() }),
  execute: (ctx) => ctx.output({ summary: ctx.args.summary, resolved: ctx.args.resolved }),
})

const concierge = app.agent({
  name: 'concierge',
  model: openai.realtime('gpt-realtime-1.5'),
  context: [app.context.system('Answer booking questions.'), app.context.history()],
  tools: [lookupStay, toBilling],
  output: endCall,
})

ctx.voice is the seam to the audio. It is a small interface on purpose — no LiveKit types cross it:

Both generateReply and say return a VoiceReply, and its one method is the one that matters on a phone call: await reply.waitForPlayout() resolves when the caller has actually heard it. Without it your next line runs while the sentence is still in the air.

Two control signals end a tool differently from a return value. ctx.output(value) makes the value the call's output and starts the hangup — that is what an output tool is, and it is what output: endCall declares. ctx.end() asks to end the call without supplying the value, and requires an output tool to exist, because the handler then runs one more forced generation to collect it. Returning a Runnable transfers instead. The outgoing agent's invocation closes with reason transferred and the incoming agent gets its own invocation_start. Its tools and instructions are rebuilt, and LiveKit swaps the agent inside the live call. The caller hears no seam.

One capability of text mode does not survive the crossing: a tool cannot yield. There is nobody to pause for — the caller is on the line — so Stopping to ask has no voice equivalent. Ask the caller instead.

Step 6

What the ledger records

A call leaves the same kind of event list a text run does, and you read it the same way. Nothing about voice is a special table.

const call = await app.sessions.get('call-4471')

const ledger = call?.events.map((event) => event.type)

// Voice writes the same `user` and `assistant` events a typed turn does, tagged with where the
// text came from.
const transcript = call?.events
  .filter((event) => event.type === 'user' || event.type === 'assistant')
  .map((event) => ({ who: event.type, said: event.text, from: event.source ?? 'text' }))

// Audio tokens ride on the same `model_end` usage a text turn reports.
const audio = call?.events
  .filter((event) => event.type === 'model_end')
  .map((event) => event.usage?.audioInputTokens)

// How the call ended: 'completed' | 'transferred' | 'inactivity_timeout' | 'max_duration' |
// 'disconnected' | 'participant_left'.
const ending = call?.events.findLast((event) => event.type === 'invocation_end')?.reason

The differences are small and worth knowing precisely:

There is no per-turn commit. Events accumulate on the session in memory as the call runs, and the handler writes them once, at the end — after the final hooks, inside LiveKit's shutdown barrier so a hangup cannot outrun it. That is the right trade for a call: a turn is a fuzzy unit here, and the caller is present throughout. It has one consequence.

Step 7

Hooks along the call

A VoiceHook is an ordinary Hook — beforeAgent, beforeTool, afterTool, onEvent, all of Guardrails and recovery — plus five callbacks that only a call has. Build one with app.hook.voice and put it in the same hooks array.

const callLifecycle = app.hook.voice({
  // Replaces the default auto-speak on entry and after every transfer.
  onEnter: async (ctx) => {
    const reply = await ctx.voice.generateReply({ instructions: 'Greet the caller by name.' })
    await reply.waitForPlayout()
  },

  // Every transcript line, on its own queue — never blocks the audio pipeline.
  onTranscript: (ctx) => {
    if (ctx.event.type === 'user') ctx.state.update({ lastHeard: ctx.event.text })
  },

  // Return false to keep the call alive, true to end it, nothing for the default end.
  onInactivity: async (ctx) => {
    if (ctx.inactivityCount >= 2) return
    const reply = await ctx.voice.generateReply({ instructions: 'Ask if they are still there.' })
    await reply.waitForPlayout()
    return false
  },

  onExpiry: async (ctx) => {
    const reply = await ctx.voice.generateReply({ instructions: 'Apologise, then wrap up.' })
    await reply.waitForPlayout()
  },

  onDisconnect: () => undefined,

  // Ephemeral telemetry. Never written to the ledger.
  onVoiceEvent: (event) => {
    if (event.type === 'forced_tool_correction' || event.type === 'voice_error') {
      console.warn('[voice]', event)
    }
  },
})

onEnter fires when an agent becomes active — at the start of the call and again after every transfer. Defining it replaces the default greeting, so the first thing the caller hears is yours. onInactivity, onExpiry and onDisconnect share one contract: return false to keep the call alive, true to end it, and nothing at all for the default. When several hooks define the same callback they all run, and any one of them returning false wins — a veto, not a vote. ctx.inactivityCount counts consecutive silences and resets the moment the caller speaks, which is what turns three prompts into an escalation instead of a loop.

onTranscript runs on its own queue, one line at a time, so a slow hook delays the lines behind it and never the audio. Its context is richer than the others'. Alongside session, state, voice and the event, it carries run(agent, input?) — so a background agent can read the conversation as it happens. A live summary, a check on what was promised, a state update the main agent never has to be told about. State written here is drained before the commit.

onVoiceEvent is the telemetry channel: state transitions, activity markers, lifecycle-hook outcomes, forced-tool corrections, errors. None of it is persisted, which is the point — it is how the call is behaving, not what happened on it. app.hook.voiceLogging({ level: 'debug' }) is a ready-made hook that formats these as structured log lines.

Step 8

Forcing a tool before speech

Sometimes one specific tool must run before the model is allowed to answer: check the policy before quoting it, record the outcome before hanging up. In text mode that is toolChoice: { name } and the provider honours it. Realtime sessions do not reliably honour a named tool choice. The obvious workaround — hide every other tool for one turn — mutates the tool list, which invalidates the provider's prompt cache. Every turn after that costs more.

So the ADK layers its own guarantee on top of the one thing providers do honour, toolChoice: 'required'. Pass a named tool choice to ctx.voice.generateReply and a gate opens for that generation:

import { ForcedToolCallError } from '@animahealth/adk/voice'

const beforeQuoting = app.hook.voice({
  onTranscript: async (ctx) => {
    if (ctx.event.type !== 'user') return
    if (!/\bcancel\b/i.test(ctx.event.text)) return

    try {
      // The gate: this generation may not end in speech until `check_policy` has run.
      await ctx.voice.generateReply({
        toolChoice: { name: 'check_policy' },
        instructions: 'Check the cancellation policy before answering.',
      })
    } catch (error) {
      if (!(error instanceof ForcedToolCallError)) throw error

      // reason: 'active_gate' | 'exhausted' | 'timeout' | 'generation_failed'
      console.warn(
        `[voice] ${error.intendedToolName} not called (${error.reason})`,
        error.incorrectToolName,
        `${error.attempts}/${error.maxAttempts}`,
      )
      await ctx.voice.say('Let me put you through to someone who can check that.')
    }
  },
})

What the gate does, in order. The named choice is downgraded to 'required' on the way to the provider — the tool list is untouched, so the cache survives. If the intended tool is called, it executes normally and the gate resolves when it finishes. If a different tool is called, that tool never executes: the call is intercepted before beforeTool and before execute, so nothing with a side effect has happened yet. The model gets a synthetic result, and a correction turn goes back naming both the tool it wrongly called and the tool it must call. If the model speaks instead of calling anything, that counts as a wrong answer too; so does calling nothing at all, after a short wait. Each of those consumes an attempt.

When the attempts run out, or the gate times out, or the generation itself fails, the gate clears and rejects. The ForcedToolCallError carries intendedToolName, incorrectToolName, attempts, maxAttempts, source and reason. The same failure is also announced on onVoiceEvent as forced_tool_failure, with forced_tool_correction for each retry along the way.

Three properties are deliberate and worth relying on. The gate is ephemeral — it lives in the voice session for one generation and is never written to session or application state, so nothing to clean up when a call drops. It is exclusive — opening a second gate while one is pending throws immediately with reason: 'active_gate' rather than interleaving two forced turns. And the correction instructions are generic: the ADK will tell the model which tool it must call and will not invent a domain-specific recovery. What to say to the caller when the tool never lands is yours, in the catch.

The same machinery runs on your behalf at the end of a call. Something asks the call to finish: ctx.end(), an inactivity or expiry hook, a caller who hung up. If the agent declares an output tool, the handler triggers one last generation forced onto it, so the structured result is collected before the room closes.

Step 9

Measuring it

A call is hard to assert on: the transcript is nondeterministic and the timing is half the product. The package's answer is app.evaluate.voice, a sibling of the runner in Measuring agents. It puts a second agent on the other end of a real LiveKit room, plays the conversation out, and scores the ledger and the timings it produced. .case, .cases and .report are shaped like their text counterparts, and a control handle can disconnect the simulated caller mid-call. It needs the same LiveKit credentials this chapter's worker does, and it is a chapter's worth of surface on its own. Start from app.evaluate.voice and the @animahealth/adk/eval exports.