Llm
Parameters
Name of the LLM model.
Name of the LLM vendor.
Webhook that will be called when the LLM session ends.
Object containing authentication credentials; format according to the model.
Object containing information such as the URI to connect to.
Webhook that will be called when a requested LLM event happens (e.g., transcript).
Array of event names listing the events requested (wildcards allowed).
Declarative transfer-to-human configuration. When present, the runtime injects a
transfer_to_human tool and runs the packaged transfer choreography
when the model calls it. See Transfer-to-human handoff.
Enable the built-in hangup tool. When present, the runtime injects a hangup tool into the model’s toolset; when the model calls it, the call is ended. Works across all supported s2s vendors. See Built-in Hangup Tool below.
Default reason placed in the X-Reason SIP header on the outbound BYE. Used as a fallback when the model does not supply its own reason at call time.
Object containing instructions for the LLM; format dependent on the LLM model.
Webhook that will be called when the LLM wants to call a function.
The following LLMs are currently supported:
- OpenAI Realtime API
- OpenAI GPT Live API (limited-access alpha)
- Deepgram Voice Agent
- Ultravox
- ElevenLabs
- Google Gemini Live API (via Gemini Developer API or Vertex AI)
- AssemblyAI Voice Agent
- xAI Voice Agent
- Qwen Omni Realtime (Alibaba Cloud Model Studio)
- Azure OpenAI Realtime
- Azure Voice Live
- Amazon Nova Sonic (Amazon Bedrock)
OpenAI Realtime
Set vendor: 'openai' and supply an OpenAI API key via auth.apiKey. llmOptions carries two payloads — response_create and session_update — that are forwarded to OpenAI as the corresponding response.create and session.update client events. The example at the top of this page uses the GA format.
Model selection
GA realtime model name. Common choices:
gpt-realtime— the standard GA realtime conversation model. Recommended starting point.gpt-realtime-2— reasoning-capable variant. Emits aphase: "commentary"message beforephase: "final_answer"in its output items; both phases produce audio, so the caller hears the model “think out loud” before the actual answer. Useful for some assistants, surprising for typical greeting-first flows.gpt-realtime-whisper— streaming transcription model. Emitsresponse.output_audio_transcript.deltaevents but does not endpoint turns server-side (no.completedin current GA); not suitable for normal speech-to-speech or gather flows.
OpenAI deprecated the Realtime preview models (gpt-4o-realtime-preview-*) on 2026-05-12; use a GA model.
Instructions
session_update.instructions sets the persona that persists for the whole session; response_create.instructions overrides it for a single response only. GA realtime models do not assume English by default — if jambonz speaks first (which it does when response.create fires before any caller audio), anchor the language in instructions (e.g. “Always respond in English unless the caller speaks another language”) or you may get a greeting in an arbitrary language.
Audio format
The audio format on the wire to OpenAI is fixed at {type: 'audio/pcm', rate: 24000}. jambonz resamples to and from the channel’s native rate (typically G.711) automatically. Any audio.input.format or audio.output.format set in the application is overridden — they are present in the example only for completeness.
Legacy preview-format compatibility
Apps written against the preview API still work without changes. jambonz detects the legacy flat format — top-level modalities, voice, input_audio_format, output_audio_format, input_audio_transcription, turn_detection, temperature — inside session_update or response_create, converts it to the GA format on the wire, and aliases GA event names back to their preview names before forwarding to your eventHook. For example, an app that subscribed to response.audio_transcript.done continues to receive events with that exact type even though the server now emits response.output_audio_transcript.done. A one-time deprecation warning is logged when the legacy format is detected.
GA-invalid fields are stripped silently with a WARN log:
temperatureinresponse_create— no longer accepted in GA.output_audio_formatas a flat string — replaced byaudio.output.format: {type, rate}(which jambonz overrides topcm24 kHz anyway).
The compatibility layer is intended as a transitional shim — plan to update applications to the GA format before it is removed.
OpenAI-specific llmOptions fields
The initial response.create.response payload sent to OpenAI when the session opens. Drives the first response from the assistant.
Array of output modalities. Typically ['audio'] for voice. (GA renamed this from preview’s modalities.)
Per-response instruction override. Replaces session-level instructions for this one response only — useful for the opening greeting. Anchor the language explicitly when the model speaks first.
Audio configuration object. Use audio.output.voice to choose a voice (e.g. alloy, marin) and audio.output.format (overridden to pcm 24 kHz by jambonz).
Maximum tokens for this response.
The initial session.update.session payload. Sets persona, tools, audio config, and VAD for the whole session.
Must be 'realtime' in GA. Required when present.
Session-wide persona prompt. Persists for every response in the session unless overridden by response_create.instructions.
Auxiliary transcription model for converting caller audio into text in the conversation log. Common values: whisper-1, gpt-4o-transcribe, gpt-realtime-whisper. This is separate from the conversation model.
Server-side VAD configuration: {type: 'server_vad', threshold, prefix_padding_ms, silence_duration_ms}. Defaults are server-supplied if omitted. type: 'semantic_vad' with an eagerness field is also supported on models that accept it.
Array of function-tool definitions exposed to the model. Each entry needs name, type: 'function', description, and JSON Schema parameters. The model calls a tool by emitting response.output_item.done with item.type: 'function_call'; jambonz routes that to your toolHook.
'auto' (default), 'none', or a specific tool name to force.
OpenAI GPT Live
Set vendor: 'gptlive' and supply an OpenAI API key in auth.apiKey to talk to OpenAI’s GPT Live API.
GPT Live requires enrollment in OpenAI’s Early Access Program. An ordinary OpenAI key completes the connection and is then refused by OpenAI at the first server event with Voice session access denied, which jambonz reports as completion_reason: 'server error'.
Expect event names and fields to change while the API is in alpha.
Not a drop-in for vendor: 'openai'
GPT Live is a different protocol from the OpenAI Realtime API, not a newer model for it. If you are moving an app over from vendor: 'openai', read Migrating from the Realtime API first — the payloads are not interchangeable.
A minimum configuration — note that on its own this produces a silent call, because nothing has asked the model to speak first (see Making the agent speak first):
A runnable version of this, with a greeting, a tool call and both delegation modes, is in the GPT Live example app.
Model selection
GPT Live model name; defaults to gpt-live-1-boulder-alpha. Set it on the verb. It must not appear inside session_update at all — jambonz rejects the verb if it does, even if you set it in only that one place.
Audio format
There is no audio format to configure. GPT Live fixes it at 24 kHz mono PCM, and jambonz converts to and from the caller’s codec for you.
session_update is required
llmOptions.session_update is not optional here: GPT Live will not accept the caller’s audio until it has your configuration. jambonz sends it first and waits for session.started — the event you should hang your greeting off, described next.
There is no response_create. Once the session starts, the model drives the conversation itself.
Making the agent speak first
Because there is no response_create, nothing tells the model to take the first turn — and putting the greeting in instructions is not enough. The model waits for the caller, who hears silence.
To open the call, send a session.context.append as soon as you receive session.started. Give it the wording you want and tell it when to speak:
Use the same pattern any time you need the agent to say something specific mid-call — a disclosure, a transfer notice, a closing.
A context append guides the model. Per OpenAI, it may paraphrase your wording, or occasionally stay silent. Omit the wording and the model will use its own.
If the exact words matter — a legal disclosure, a brand greeting — do not ask the model. Play it yourself with a say verb before the llm verb.
Requesting a greeting needs WebSocket transport. The llm:update command is not accepted by the REST updateCall API, so a webhook-only application has no way to send one.
Delegation: how the model asks for work
Delegation is GPT Live’s distinguishing concept, and it has no direct analogue in the other vendors on this page. Anything the model needs from outside the spoken conversation — a fact it does not know, a function it wants run — arrives as a delegation. Your choice of kind determines whether you can use tools at all.
Omit it entirely and the model simply never asks for anything — no tools, no context requests. That is the simplest way to start.
'client' — the model asks your application for background information in prose. Simple to implement, but there is no function calling.
'responses' — the model runs a turn on OpenAI’s Responses API that can call your functions. Required if you want tools, mcpServers, handoff or hangup.
Responses delegation (function calling)
Declare your tools inside the delegation. Note that responses.model is a second, separate model that runs the delegated turn — it is not the voice model on the verb:
Tools go in delegation.responses.tools — not delegation.tools — and both the nested responses object and its model are required.
jambonz rejects the verb before the call connects if you configure handoff, hangup or mcpServers without them. It cannot check your own tool declarations, so if you omit responses.model there, OpenAI refuses the session at startup and you get completion_reason: 'server error'.
Client delegation (supplying context)
The model emits delegation.created with item.target: 'client'; you answer with the item’s id and up to 500 tokens of prose in a single input_text part:
Always answer a client delegation. There is no way to cancel one, so if you leave it unanswered the model waits and the caller hears silence.
Handling tool calls
With a responses delegation, jambonz calls your toolHook with {name, args, tool_call_id} just as it does for every other vendor. What differs is the envelope you send the result back in:
output must be a string. Send one of these per call if the model asked for several, and do not follow it with a response.create — unlike the Realtime API, the server resumes on its own.
The built-in handoff and hangup tools, and any mcpServers you configure, all work here too — but only with a responses delegation, since a client delegation has no way to call a function.
Following the conversation
GPT Live is in alpha and has no public event reference, so this is the list. Name the ones you want in the verb’s events array — wildcards such as response.* work, as do 'all' plus -eventName exclusions.
If you omit events entirely, jambonz forwards everything — including transcript fragments, which are high volume. Name the events you need.
Prefer turn.done over the *_transcript.added fragments for following the conversation: fragment boundaries follow speech cadence rather than complete thoughts, so one sentence may arrive in several pieces.
Barge-in needs no work on your part — jambonz detects it and flushes the queued agent audio; the underlying signal is not forwarded to your eventHook.
Updating the session mid-call
An llm:update command (session.updateLlm() in the Node SDK) accepts these five events:
session.update— change configuration. Sparse: fields you omit keep their values. Replacingdelegationrequires the whole object.session.context.append— add context or request a spoken messagedelegation.context.append— answer a client delegationdelegation.function_call_output.create— return a tool resultsession.close— end the session gracefully
Anything else is discarded silently — you will not get an error back, so check the type if an update appears to do nothing.
How the session ends
The actionHook fires with a completion_reason:
A handoff that bridges reports the transfer outcome instead. Unlike some other vendors, GPT Live never reports server failure.
Not every problem ends the call. Once the session is running, a rejected client event — a stale delegation_item_id, an over-long context append — or a failed delegation is reported to your eventHook and the conversation continues. Handle those events if you want to recover; ignore them and the model continues without the context or tool result you were sending.
cancelOnBargeIn and cancelOnResponseTimeout have no effect with this vendor, because GPT Live has nothing to cancel. responseTimeoutMs does work, but it is disabled by default — set it to a non-zero value and you get a response.timeout event if a delegation stalls, which you can recover from with an llm:update.
Migrating from the Realtime API
If you are porting an app from vendor: 'openai':
Troubleshooting
xAI Voice Agent
Set vendor: 'xai' and supply an xAI API key via auth.apiKey. xAI’s Voice Agent speaks the same OpenAI Realtime GA dialect described in the OpenAI Realtime section above — llmOptions carries the same response_create and session_update payloads, in the same GA format, forwarded as the response.create and session.update client events. jambonz connects to wss://api.x.ai/v1/realtime.
Model selection
xAI Voice Agent model name.
grok-voice-latest— default; currently aliases togrok-voice-think-fast-1.0.grok-voice-think-fast-1.0— flagship model.
Voices
Set the voice via session_update.audio.output.voice (or response_create.audio.output.voice), same field as OpenAI. Available voices: eve (default), ara, rex, sal, leo.
Required session_update
Unlike OpenAI, llmOptions.session_update is required for xai — audio does not begin flowing until after the first session.updated server event is received.
Audio format
Audio on the wire to xAI is pcm16 at 24 kHz. jambonz forces this rate/format regardless of what is set in session_update.audio; do not attempt to select a different format or sample rate.
Turn detection
Unlike OpenAI GA, which nests turn detection under session_update.audio.input.turn_detection, xAI expects turn_detection top-level in session_update — i.e. session_update.turn_detection, not session_update.audio.input.turn_detection. { type: 'server_vad' } is the common setting. If turn_detection is omitted or set to null (manual turn mode), the application is responsible for turn-taking and can send input_audio_buffer.commit / input_audio_buffer.clear client events via an llm:update command.
Tool calls
Tool/function calling uses the same fields as OpenAI’s (session_update.tools, tool_choice). The completed tool call arrives on the response.function_call_arguments.done event rather than OpenAI’s response.output_item.done; jambonz routes it to your toolHook the same way.
Input transcription
When session_update.audio.input.transcription is configured, caller-speech transcripts arrive on conversation.item.input_audio_transcription.updated. Unlike OpenAI’s .completed event, this event is cumulative — each update contains the full transcript so far, not just a delta.
Qwen Omni Realtime
Set vendor: 'qwen' to use Alibaba’s Qwen Omni Realtime models on Alibaba Cloud Model Studio (DashScope). Qwen speaks an OpenAI Realtime-style dialect: llmOptions carries a session_update and an optional response_create, forwarded as the session.update and response.create client events. jambonz connects to wss://{host}/api-ws/v1/realtime?model={model}.
Model selection
Qwen Omni Realtime model name, for example qwen3.8-omni-flash-realtime, qwen3.5-omni-plus-realtime or qwen3.5-omni-flash-realtime. Defaults to qwen3.5-omni-flash-realtime.
Authentication and host
A DashScope API key, sent as a bearer token. Keys are bound to a region and workspace, so the host must match the key.
The host the key belongs to: a workspace host such as ws-xxxx.ap-southeast-1.maas.aliyuncs.com, or dashscope.aliyuncs.com for the China (Beijing) region. Defaults to the international endpoint, dashscope-intl.aliyuncs.com. A pasted console URL works too; jambonz keeps only the hostname. connectOptions.host takes precedence when both are set.
Required session_update
llmOptions.session_update is required. Audio does not flow until Qwen confirms the first session.updated.
Speaking first
response_create is optional. With server-side turn detection Qwen replies on its own once the caller speaks, so omit response_create to have the agent wait for the caller. Include one to have it speak first, for example with a greeting in its instructions.
Audio format
jambonz sends Qwen the caller’s audio as 16 kHz pcm16 and plays Qwen’s 24 kHz audio to the caller. Do not set a different audio format in session_update.
Input transcription
Qwen transcribes the caller only when asked. Set session_update.input_audio_transcription (for example { model: 'qwen3-asr-flash-realtime' }); the caller’s words then arrive on conversation.item.input_audio_transcription.completed.
Tool calls
Tools go in session_update.tools, in the same function format as OpenAI. jambonz routes each call to your toolHook and returns its result to Qwen.
Barge-in
When the caller talks over the agent, jambonz cancels the reply in progress if the verb’s cancelOnBargeIn is set, and only when it is a reply jambonz asked for.
Azure OpenAI Realtime
Set vendor: 'microsoft' to use OpenAI’s realtime models deployed on your own Azure OpenAI resource. It speaks the same OpenAI Realtime GA dialect described in the OpenAI Realtime section above: llmOptions carries the same response_create and session_update payloads, in the same format. jambonz connects to Azure’s GA realtime endpoint, wss://{host}/openai/v1/realtime?model={deployment}.
Model and connection
The name of your Azure deployment of a realtime model, which is not necessarily the model’s name.
Your Azure OpenAI resource host, for example my-resource.openai.azure.com.
Overrides the endpoint path, for example to reach Azure’s preview realtime endpoint instead of the GA one.
A key of the Azure OpenAI resource. jambonz sends it as the api-key query parameter.
Speaking first
As with OpenAI, llmOptions.response_create is required: jambonz sends it as soon as the session opens, so the agent always speaks first.
Input transcription
Azure transcribes the caller only with a transcription model deployed on the same resource. Set session_update.audio.input.transcription.model to that deployment’s name; without one, no caller transcripts arrive.
vendor: 'microsoft' is not Azure Voice Live, which is a separate Azure service with its own session shape.
Azure Voice Live
Set vendor: 'voicelive' and point connectOptions.host at your Microsoft Foundry or Azure Speech resource to use the Azure Voice Live API. jambonz connects to wss://{host}/voice-live/realtime.
Voice Live is not the same thing as vendor: 'microsoft'. That vendor targets a realtime model deployed on your Azure OpenAI resource (see Azure OpenAI Realtime) and shares its payload shapes with vendor: 'openai'. Voice Live is a separate service that keeps the Realtime event vocabulary but uses its own flat session shape and adds Azure-only capabilities: Azure Speech voices, semantic VAD, server-side noise suppression and echo cancellation, word timestamps and visemes, and non-realtime chat models (e.g. gpt-4.1) paired with Azure speech to text.
Connection options
Your resource hostname, e.g. my-resource.services.ai.azure.com (or my-resource.cognitiveservices.azure.com for older resources).
Voice Live api-version. Defaults to 2026-04-10.
Foundry Agent Service agent id. When set, the session is addressed by agent (agent_id, plus projectId as project_id) instead of by model.
Authentication
Supply either auth.apiKey, which jambonz sends as the api-key query parameter, or auth.accessToken, a Microsoft Entra ID token sent as an Authorization: Bearer header. Entra tokens are short-lived, so mint one per call in your webhook; the token must be issued for the https://ai.azure.com/.default scope.
Required session_update
llmOptions.session_update is required — caller audio does not flow until the first session.updated server event arrives.
It is sent verbatim, in Voice Live’s flat shape: voice, turn_detection, modalities and input_audio_transcription sit at the top level of session, not nested under audio.input / audio.output the way OpenAI’s GA Realtime format has them. Do not port an openai_s2s session_update across unchanged.
Voices
Set voice as an object, not a string: { name, type }, where type is azure-standard (Azure Neural, HD and MAI voices), azure-custom, or azure-realtime-native (the curated voice set for the azure-realtime model). HD voices accept an optional temperature, and any Azure voice accepts rate between 0.5 and 1.5.
Turn detection
Beyond server_vad, Voice Live offers azure_semantic_vad and azure_semantic_vad_multilingual, which decide end-of-turn from meaning rather than volume and can drop filler words (remove_filler_words: true) so an “umm” does not trigger barge-in. These work with every model, unlike OpenAI’s semantic_vad.
Audio format
Audio on the wire is pcm16 at 24 kHz, which matches Voice Live’s default input_audio_sampling_rate. jambonz forces this rate; do not set input_audio_sampling_rate: 16000.
Input transcription
Azure speech to text is active automatically only with a non-multimodal model (e.g. gpt-4.1). A native-audio model such as gpt-realtime-2.1 emits no conversation.item.input_audio_transcription.completed events at all unless you ask for transcription explicitly:
With a non-multimodal model, Azure speech to text is active automatically. Set input_audio_transcription.model to choose another: azure-speech, mai-transcribe, or — with gpt-realtime / gpt-realtime-mini — whisper-1, gpt-4o-transcribe, gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize.
Voice Live-only events
response.audio_timestamp.delta / .done (word timestamps, enabled with output_audio_timestamp_types: ['word']) and response.animation_viseme.delta / .done (enabled with animation: {outputs: ['viseme_id']}) are forwarded to your eventHook alongside the standard Realtime events.
Voice Live’s text to speech avatar requires a separate WebRTC SDP exchange with the service, which has no place in a SIP call. Setting avatar in session_update will not produce video.
Amazon Nova Sonic
Amazon Nova Sonic is Amazon’s speech-to-speech model: it listens to the caller and answers in its own voice, with no separate speech recognition or text-to-speech step. Nova 2.5 Sonic, generally available since October 2026, improves on Nova 2 Sonic’s reasoning, instruction following and tool-calling accuracy at the same price, and its tool calls are asynchronous, so the agent keeps talking with the caller while your toolHook runs.
Set vendor: 'nova' to use Amazon Nova Sonic (Nova 2 Sonic and Nova 2.5 Sonic) on Amazon Bedrock. Nova Sonic is not reached over a websocket: the media server opens a Bedrock bidirectional stream (InvokeModelWithBidirectionalStream) signed with the AWS credentials you supply. This vendor requires the mediajam media server.
Model selection
model is the Bedrock model id and defaults to amazon.nova-2-5-sonic. amazon.nova-2-sonic-v1:0 also works. Nova Sonic is available in us-east-1, us-west-2, eu-north-1 and ap-northeast-1.
Authentication
auth.accessKeyId and auth.secretAccessKey are required; add auth.sessionToken for temporary STS credentials. auth.region defaults to us-east-1. The IAM principal needs bedrock:InvokeModelWithBidirectionalStream on the model. jambonz never falls back to the server’s own AWS identity.
llmOptions
System instructions, sent as Nova’s SYSTEM text content.
Sent right after the system prompt with role SYSTEM_SPEECH. It controls the script Nova uses for Hindi transcription and responses, and accepts only one of the three fixed sentences in the Nova system prompt guide (Latin, Devanagari or mixed script).
Prior turns as {role: 'user' | 'assistant', content}, sent before the audio starts. jambonz applies Nova’s rules: consecutive turns with the same role are merged, leading assistant turns are dropped (history must start with the user), and the total is capped at 200KB by dropping the oldest turns.
Nova waits for the caller to speak first. Set a greeting to send a user text turn as soon as the audio starts, so the agent opens the conversation.
Merged into Nova’s sessionStart event: inferenceConfiguration (maxTokens, topP, temperature) and turnDetectionConfiguration.endpointingSensitivity (HIGH, MEDIUM or LOW).
Merged into Nova’s promptStart event. Use audioOutputConfiguration.voiceId to pick a voice (default matthew), toolConfiguration.tools for tools in Nova’s toolSpec shape (inputSchema.json is a JSON string), and toolConfiguration.toolChoice to steer tool use: {auto: {}} (default), {any: {}} or {tool: {name: 'get_weather'}}. In our testing amazon.nova-2-sonic-v1:0 honors a forced tool, while amazon.nova-2-5-sonic accepts toolChoice but does not force the call. jambonz sets the promptName and the audio format.
Nova has no keypad input. Set dtmf: {enable: true, ...} and jambonz collects the caller’s digits using the gather verb’s options (finishOnKey, numDigits, minDigits, maxDigits, interDigitTimeout, timeout) and sends each completed entry to Nova as a user text turn.
A system prompt or any other text over Nova’s 50KB per-event limit is split across several textInput events for you.
Voices
tiffany and matthew are polyglot: they speak all of Nova’s languages, so the voice stays the same if the caller switches language.
Audio format
Caller audio is sent to Nova as 16 kHz pcm16 and Nova’s speech is received as 24 kHz pcm16. jambonz sets both rates, so a sampleRateHertz in promptStart is overridden.
Tool calls
MCP tools and the handoff and hangup tools are added to toolConfiguration.tools for you. When Nova emits a toolUse event, jambonz calls your toolHook with tool_call_id (Nova’s toolUseId), name and args. Reply with any JSON object; it is returned to Nova as the tool result. To report a failure, return something like {error: 'order not found'}. Nova waits until every tool call has a result, so if no toolHook is configured jambonz answers the call with an error result rather than leaving it open. Tool calls are asynchronous: Nova can keep talking while your hook runs.
Events
The eventHook receives Nova’s own events, {event: {<name>: {...}}}: completionStart, contentStart, textOutput (the caller’s transcript and the agent’s text), toolUse, contentEnd, completionEnd, usageEvent, userSpeechStart / userSpeechEnd (sent by Nova 2.5 when the caller starts and stops speaking) and error, plus jambonz’s output_audio.playback_started / output_audio.playback_stopped. Filter with events using these names. As with every jambonz hook payload, keys arrive snake_cased: {event: {text_output: {content_id, role, content}}}, content_start, additional_model_fields and so on.
Updating the session mid-call
Send llm:update with {text: '...'} to inject a user turn as text (Nova answers it in speech), or with a raw {event: {...}}, which is forwarded with the session’s promptName filled in.
Long calls
Bedrock ends each bidirectional stream after 8 minutes. jambonz renews the stream before then, at a moment when the agent is not speaking. It replays the setup and the conversation so far as text history, so the call continues without a break. A caller speaking at the exact moment of renewal may need to repeat themselves.
ElevenLabs Agents
Set vendor: 'elevenlabs' to connect the caller to an ElevenLabs agent. The agent’s prompt, voice and tools are configured in ElevenLabs, so model is not used. Identify the agent with auth.agent_id; add auth.api_key to connect with a signed URL, which ElevenLabs requires when the agent has authentication enabled.
Caller keypad input (DTMF)
Set llmOptions.dtmf.enable to pass the caller’s key presses to the agent. jambonz collects the digits into one entry and sends it as a single user message, so the agent receives 12345, not five separate turns. The digits are the whole message; tell the agent in its prompt what keypad entries mean (a ZIP code, a menu choice).
An entry ends the same way digit collection ends in gather, with the same option names:
Turns keypad input on. Off by default.
A key that ends the entry, such as #. The key itself is not sent. No default: without it, # and * are sent as ordinary digits.
Send the entry as soon as this many digits have been pressed.
Maximum digits in one entry; the entry is sent when it is reached.
Minimum digits before interDigitTimeout or timeout can end the entry. Defaults to 1. An entry shorter than this when timeout expires is discarded.
Seconds of silence after the last key that end the entry. No default.
Seconds from the first key until the entry is sent, if nothing else has ended it. Defaults to 15; 0 turns it off. Once interDigitTimeout is running it takes over, so a caller who keeps typing is not cut off.
With only enable: true, an entry ends after timeout, 15 seconds after the first key, so a one-key menu choice reaches the agent 15 seconds late. Set interDigitTimeout (2 seconds works well), finishOnKey, or numDigits to match what the caller is entering.
Each key press also tells the agent the caller is active, so its turn timeout does not fire while the caller is typing. If the agent is still speaking when the entry is sent, it stops, as it would if the caller spoke.
The agent’s own DTMF input settings in ElevenLabs (digit timeout and # terminator) apply only to ElevenLabs’ telephony integrations; on a jambonz call, use the options above instead. You do not need to enable DTMF input on the agent.
Updating the conversation mid-call
An llm:update command (session.updateLlm() in the Node SDK) accepts these ElevenLabs client events:
user_message— send text as a user turn, for example{type: 'user_message', text: 'My order number is 4471'}contextual_update— give the agent background information without interrupting it, for example{type: 'contextual_update', text: 'The caller is a premium customer'}user_activity— tell the agent the caller is still there, which resets its turn timeout
user_message and contextual_update need a text string. Anything else is discarded, and you will not get an error back.
llm:update needs WebSocket transport. The REST updateCall API does not accept it.
Google Gemini Live
Set vendor: 'google' to use Gemini’s Live API. The same vendor: 'google' integration reaches the Live API through two access paths:
Both speak the same JSON wire protocol — setup, realtimeInput, serverContent, modelTurn, toolCall, sessionResumptionUpdate are identical. Only the URL, auth, and model resource format differ. llmOptions.setup is forwarded verbatim to Google’s BidiGenerateContentSetup message after the WebSocket connects.
Gemini Developer API (API key)
Default access path. Provide an API key via auth.apiKey; the key starts with AIza. Model name uses the short form models/<id>. connectOptions is optional — defaults to the Gemini Developer endpoint.
Vertex AI Live API (OAuth Bearer)
Production access path. The application mints an OAuth access token from a Google Cloud service-account JSON key and passes the token string (starting with ya29.) as auth.apiKey. jambonz detects the ya29. prefix and sends it as an Authorization: Bearer … header on the WebSocket upgrade. Model name must be the full Vertex resource path.
Setup checklist for Vertex AI:
- GCP project with
aiplatform.googleapis.comAPI enabled and billing attached. - Service account with the
roles/aiplatform.userIAM role on the project. - Service-account JSON key file accessible to the application (store outside repo, reference via env var).
- Host region prefix in
connectOptions.hostmust match thelocations/<region>segment inmodel. e.g.us-central1-aiplatform.googleapis.comwithprojects/.../locations/us-central1/.... Mismatched regions returnPublisher Model not found.
OAuth token lifetime: tokens issued from a service-account JWT expire after ~1 hour. jambonz uses whatever token the application provides at session start; it is not refreshed mid-session. The WebSocket stays open across expiry, but a reconnect with a stale token will fail. For typical call durations this is fine; mint a fresh token per call.
Google-specific connectOptions fields
WebSocket host. Default: generativelanguage.googleapis.com (Gemini Developer). For Vertex AI use {LOCATION}-aiplatform.googleapis.com matching the region in your model resource path.
WebSocket path. Default: /ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent (Gemini Developer). For Vertex AI use /ws/google.cloud.aiplatform.v1beta1.LlmBidiService/BidiGenerateContent.
Legacy — Gemini Developer API version string (v1, v1beta, v1alpha). Ignored when path is set. Kept for backward compatibility.
Google-specific llmOptions fields
The BidiGenerateContentSetup object sent to Gemini right after the websocket connects. The model field is populated automatically from the verb’s model parameter. generationConfig.responseModalities is forced to audio.
Optional proactive greeting. When set, jambonz sends a text message to Gemini immediately after setup so the agent speaks first without waiting for the caller to speak. Accepts either a string or an object with a text field. The value is an instruction to the model, not the literal words — for example "Greet the caller warmly" rather than "Hello, how can I help?".
Implemented using realtimeInput.text so it works on both the 2.0 Live models and gemini-3.1-flash-live-preview. (On 3.1, clientContent is reserved for seeding history and does not trigger a model response, which is why realtimeInput.text is used.)
Enable session resumption. Pass {} to opt in, or { handle: "..." } to resume a previous session. Resumption handles are delivered back to the application via llm_event sessionResumptionUpdate messages.
Tool calls
Declare your functions in llmOptions.setup.tools, as Gemini’s functionDeclarations. MCP tools and the handoff and hangup tools are added for you. When Gemini calls a function, jambonz calls your toolHook. The payload includes function_calls, Gemini’s list of calls (its functionCalls, snake_cased like every hook payload), each with its id, name and args.
Gemini expects each result back in its own toolResponse message: a functionResponses entry with the call’s id and name, and your result as response.
Gemini waits until every call has a response, so answer each one, with an error result if it failed (for example { error: 'order not found' }).
Send the result inside toolResponse.functionResponses. Releases before 11.1.12 drop a bare result such as { ok: true }, logging invalid tool output, must be functionResponses. Gemini never gets the answer, and the agent stalls. From 11.1.12, jambonz wraps a bare JSON object for you, adding the call’s id and name, and sends each function call to your toolHook separately with name, args and tool_call_id. The toolResponse form works in every release.
AssemblyAI Voice Agent
Set vendor: 'assemblyai' and supply your AssemblyAI API key via auth.api_key. llmOptions is the AssemblyAI Voice Agent session payload passed through verbatim — there is no jambonz-specific wrapper. jambonz wraps it as {type: 'session.update', session: <llmOptions>} and sends it as the first client message after the websocket connects to wss://agents.assemblyai.com/v1/ws. See the AssemblyAI Voice Agent product page and Voice Agent API docs for an overview.
The audio format is not configurable. AssemblyAI Voice Agent only accepts audio/pcm at 24 kHz, which jambonz uses unconditionally — session.input.format / session.output.format set by the application are overridden. jambonz resamples to/from the channel’s native rate automatically.
AssemblyAI-specific auth fields
Your AssemblyAI API key. Sent as Authorization: Bearer <api_key> on the WebSocket handshake.
AssemblyAI-specific llmOptions fields
AssemblyAI’s protocol requires a session.update message, but every field inside is optional — pass llmOptions: {} to start with all server defaults.
System prompt for the agent.
Initial greeting the agent will speak when the session opens.
Output audio configuration. Supports voice — see the AssemblyAI voices reference for available IDs. The format sub-field is overridden by jambonz.
Input audio configuration. Supports keyterms (array of biasing terms) and turn_detection (vad_threshold, min_silence, max_silence, interrupt_response). The format sub-field is overridden by jambonz.
Array of tool definitions. Each entry must include type: "function", name, description, and parameters (JSON Schema). jambonz auto-fills type: "function" if omitted.
Tool calls
The agent invokes a tool by emitting a tool.call server event. jambonz routes it to the application’s toolHook with {name, args, tool_call_id}. The application replies via session.sendToolOutput(tool_call_id, {type: 'tool.result', tool_call_id, result}). The result should be a string (JSON-stringify objects before sending) — jambonz JSON-stringifies non-string result values automatically.
Transfer-to-human handoff
Add a handoff block to let the realtime model transfer the caller to a human. The runtime
injects a transfer_to_human tool — no toolHook is needed for it. When the caller asks
for a human and the model calls the tool, jambonz runs the packaged
transfer choreography to the configured destination.
The handoff block accepts every transfer option (mode,
target, blindMethod, disposition, confirm, etc.), plus brief ('auto' | 'none'
| { template }), briefSynthesizer, toolName, and toolDescription — see
the agent verb’s handoff section for the details,
which apply identically here. When the human leg bridges, the actionHook reports the
transfer outcome.
For vendor: 'gptlive' the injected transfer_to_human tool needs a function-calling channel, so
llmOptions.session_update.delegation.type must be 'responses'; the verb is rejected otherwise, rather
than silently leaving the caller with no way to reach a human — see OpenAI GPT Live.
Built-in Hangup Tool
Set hangup to let the model end the call on its own. The runtime injects a hangup tool into the model’s toolset; you do not define it in llmOptions and you do not handle it in your toolHook — the runtime intercepts the call, hangs up, and ends the llm verb. This works the same way across every supported s2s vendor (OpenAI, Deepgram Voice Agent, Ultravox, ElevenLabs, Google Gemini Live, AssemblyAI, xAI Voice Agent, Azure Voice Live, OpenAI GPT Live — the last requires delegation.type: 'responses').
The injected tool accepts an optional reason argument that the model may fill in when it decides to end the call. The reason placed in the X-Reason SIP header on the outbound BYE is resolved as follows:
- The model-supplied
reasonargument, if present. - Otherwise the app-supplied
hangup.reasondefault, if configured. - Otherwise no
X-Reasonheader is sent.
Pass an empty object (hangup: {}) to enable the tool with no default reason.
When the model calls the hangup tool, the call is released immediately. The actionHook still fires, but any follow-on verbs it returns are discarded because the call is already being torn down.
For ElevenLabs, tools are configured on the ElevenLabs agent rather than injected per-session; configure a client tool named hangup on your agent and jambonz will intercept it and end the call.
Example Applications
Please checkout the following example applications:
- for OpenAI
- for Deepgram
- for Ultravox
- for ElevenLabs
- for Google Gemini (Developer API + Vertex AI)
- for AssemblyAI
- for OpenAI GPT Live