Llm

1
session.llm({
2
  vendor: 'openai',
3
  model: 'gpt-realtime',
4
  auth: { apiKey },
5
  actionHook: '/final',
6
  eventHook: '/event',
7
  toolHook: '/toolCall',
8
  events: [
9
    'conversation.item.*',
10
    'response.output_audio_transcript.done',
11
    'input_audio_buffer.committed'
12
  ],
13
  llmOptions: {
14
    response_create: {
15
      output_modalities: ['audio'],
16
      instructions: 'Greet the caller warmly in English and ask how you can help today.',
17
      audio: {
18
        output: {
19
          voice: 'alloy',
20
          format: { type: 'audio/pcm', rate: 24000 }
21
        }
22
      },
23
      max_output_tokens: 4096
24
    },
25
    session_update: {
26
      type: 'realtime',
27
      instructions:
28
        'You are a friendly, helpful voice assistant on a phone call. ' +
29
        'Always respond in English unless the caller explicitly speaks another language. ' +
30
        'Keep responses concise and natural for spoken conversation. ' +
31
        'If asked about weather, call the get_weather function.',
32
      tools: [
33
        {
34
          name: 'get_weather',
35
          type: 'function',
36
          description: 'Get the weather at a given location',
37
          parameters: {
38
            type: 'object',
39
            properties: {
40
              location: {
41
                type: 'string',
42
                description: 'Location to get the weather from',
43
              },
44
              scale: {
45
                type: 'string',
46
                enum: ['fahrenheit', 'celsius'],
47
              },
48
            },
49
            required: ['location', 'scale'],
50
          },
51
        },
52
      ],
53
      tool_choice: 'auto',
54
      audio: {
55
        input: {
56
          format: { type: 'audio/pcm', rate: 24000 },
57
          transcription: { model: 'whisper-1' },
58
          turn_detection: {
59
            type: 'server_vad',
60
            threshold: 0.8,
61
            prefix_padding_ms: 300,
62
            silence_duration_ms: 500
63
          }
64
        },
65
        output: {
66
          format: { type: 'audio/pcm', rate: 24000 },
67
          voice: 'alloy'
68
        }
69
      }
70
    }
71
  }
72
});

Parameters

The following LLMs are currently supported:

• OpenAI Realtime API
• OpenAI GPT Live API (limited-access alpha)
• Deepgram Voice Agent
• Ultravox
• ElevenLabs
• Google Gemini Live API (via Gemini Developer API or Vertex AI)
• AssemblyAI Voice Agent
• xAI Voice Agent

OpenAI Realtime

Set vendor: 'openai' and supply an OpenAI API key via auth.apiKey. llmOptions carries two payloads — response_create and session_update — that are forwarded to OpenAI as the corresponding response.create and session.update client events. The example at the top of this page shows the GA-format shape.

Model selection

Instructions

session_update.instructions sets the persona that persists for the whole session; response_create.instructions overrides it for a single response only. GA realtime models do not assume English by default — if jambonz speaks first (which it does when response.create fires before any caller audio), anchor the language in instructions (e.g. “Always respond in English unless the caller speaks another language”) or you may get a greeting in an arbitrary language.

Audio format

The audio format on the wire to OpenAI is fixed at {type: 'audio/pcm', rate: 24000}. jambonz resamples to and from the channel’s native rate (typically G.711) automatically. Any audio.input.format or audio.output.format set in the application is overridden — they are present in the example only for completeness.

Legacy preview-shape compatibility

Apps written against the preview API still work without changes. jambonz detects the legacy flat shape — top-level modalities, voice, input_audio_format, output_audio_format, input_audio_transcription, turn_detection, temperature — inside session_update or response_create, converts it to the GA shape on the wire, and aliases GA event names back to their preview names before forwarding to your eventHook. For example, an app that subscribed to response.audio_transcript.done continues to receive events with that exact type even though the server now emits response.output_audio_transcript.done. A one-time deprecation warning is logged when the legacy shape is detected.

GA-invalid fields are stripped silently with a WARN log:

• temperature in response_create — no longer accepted in GA.
• output_audio_format as a flat string — replaced by audio.output.format: {type, rate} (which jambonz overrides to pcm 24 kHz anyway).

The compatibility layer is intended as a transitional shim — plan to update applications to the GA shape before it is removed.

OpenAI-specific llmOptions fields

OpenAI GPT Live

Set vendor: 'gptlive' and supply an OpenAI API key in auth.apiKey to talk to OpenAI’s GPT Live API.

A minimum configuration — note that on its own this produces a silent call, because nothing has asked the model to speak first (see Making the agent speak first):

1
session.llm({
2
  vendor: 'gptlive',
3
  model: 'gpt-live-1-boulder-alpha',
4
  auth: { apiKey: process.env.OPENAI_API_KEY },
5
  actionHook: '/final',
6
  eventHook: '/event',
7
  toolHook: '/toolCall',
8
  llmOptions: {
9
    session_update: {
10
      instructions:
11
        'You are a friendly, helpful voice assistant on a phone call. ' +
12
        'Always respond in English unless the caller explicitly speaks another language. ' +
13
        'Keep responses concise and natural for spoken conversation.',
14
      audio: {
15
        output: { voice: 'marin' }
16
      }
17
    }
18
  }
19
});

A runnable version of this, with a greeting, a tool call and both delegation modes, is in the GPT Live example app.

Model selection

Audio format

There is no audio format to configure. GPT Live fixes it at 24 kHz mono PCM, and jambonz converts to and from the caller’s codec for you.

session_update is required

llmOptions.session_update is not optional here: GPT Live will not accept the caller’s audio until it has your configuration. jambonz sends it first and waits for session.started — the event you should hang your greeting off, described next.

There is no response_create. Once the session starts, the model drives the conversation itself.

Making the agent speak first

Because there is no response_create, nothing tells the model to take the first turn — and putting the greeting in instructions is not enough. The model waits for the caller, who hears silence.

To open the call, send a session.context.append as soon as you receive session.started. Give it the wording you want and tell it when to speak:

1
session.on('/event', (evt) => {
2
  if (evt.type === 'session.started') {
3
    session.updateLlm({
4
      type: 'session.context.append',
5
      content: [{
6
        type: 'input_text',
7
        text: 'Immediately greet the caller using the exact text below. Do not wait for the '
8
          + 'caller to speak first. After the greeting, pause and listen.\n\n'
9
          + 'Hi, I am your support assistant. How can I help you today?',
10
      }],
11
    });
12
  }
13
});

Use the same pattern any time you need the agent to say something specific mid-call — a disclosure, a transfer notice, a closing.

Delegation: how the model asks for work

Delegation is GPT Live’s distinguishing concept, and it has no direct analogue in the other vendors on this page. Anything the model needs from outside the spoken conversation — a fact it does not know, a function it wants run — arrives as a delegation. Your choice of kind determines whether you can use tools at all.

Responses delegation (function calling)

Declare your tools inside the delegation. Note that responses.model is a second, separate model that runs the delegated turn — it is not the voice model on the verb:

1
delegation: {
2
  type: 'responses',
3
  responses: {
4
    model: 'gpt-5.5',          // required
5
    tools: [
6
      {
7
        type: 'function',
8
        name: 'get_weather',
9
        description: 'Get the current weather for a city.',
10
        parameters: {
11
          type: 'object',
12
          properties: { location: { type: 'string' } },
13
          required: ['location']
14
        }
15
      }
16
    ]
17
  }
18
}

Client delegation (supplying context)

The model emits delegation.created with item.target: 'client'; you answer with the item’s id and up to 500 tokens of prose in a single input_text part:

1
session.on('/event', (evt) => {
2
  if (evt.type === 'delegation.created' && evt.item.target === 'client') {
3
    session.updateLlm({
4
      type: 'delegation.context.append',
5
      delegation_item_id: evt.item.id,
6
      content: [{ type: 'input_text', text: 'It is 62 degrees and raining in Seattle.' }]
7
    });
8
  }
9
});

Always answer a client delegation. There is no way to cancel one, so if you leave it unanswered the model waits and the caller hears silence.

Handling tool calls

With a responses delegation, jambonz calls your toolHook with {name, args, tool_call_id} just as it does for every other vendor. What differs is the envelope you send the result back in:

1
session.on('/toolCall', async(evt) => {
2
  const { tool_call_id, name, args } = evt;
3
4
  session.sendToolOutput(tool_call_id, {
5
    type: 'delegation.function_call_output.create',
6
    item: {
7
      type: 'function_call_output',
8
      call_id: tool_call_id,
9
      output: JSON.stringify({ temperature: 62, conditions: 'rain' })
10
    }
11
  });
12
});

output must be a string. Send one of these per call if the model asked for several, and do not follow it with a response.create — unlike the Realtime API, the server resumes on its own.

The built-in handoff and hangup tools, and any mcpServers you configure, all work here too — but only with a responses delegation, since a client delegation has no way to call a function.

Following the conversation

GPT Live is in alpha and has no public event reference, so this is the list. Name the ones you want in the verb’s events array — wildcards such as response.* work, as do 'all' plus -eventName exclusions.

Prefer turn.done over the *_transcript.added fragments for following the conversation: fragment boundaries follow speech cadence rather than complete thoughts, so one sentence may arrive in several pieces.

Barge-in needs no work on your part — jambonz detects it and flushes the queued agent audio; the underlying signal is not forwarded to your eventHook.

Updating the session mid-call

An llm:update command (session.updateLlm() in the Node SDK) accepts these five events:

• session.update — change configuration. Sparse: fields you omit keep their values. Replacing delegation requires the whole object.
• session.context.append — add context or request a spoken message
• delegation.context.append — answer a client delegation
• delegation.function_call_output.create — return a tool result
• session.close — end the session gracefully

Anything else is discarded silently — you will not get an error back, so check the type if an update appears to do nothing.

How the session ends

The actionHook fires with a completion_reason:

A handoff that bridges reports the transfer outcome instead. Unlike some other vendors, GPT Live never reports server failure.

Not every problem ends the call. Once the session is running, a rejected client event — a stale delegation_item_id, an over-long context append — or a failed delegation is reported to your eventHook and the conversation continues. Handle those events if you want to recover; ignore them and the model continues without the context or tool result you were sending.

Migrating from the Realtime API

If you are porting an app from vendor: 'openai':

Troubleshooting

xAI Voice Agent

Set vendor: 'xai' and supply an xAI API key via auth.apiKey. xAI’s Voice Agent speaks the same OpenAI Realtime GA dialect described in the OpenAI Realtime section above — llmOptions carries the same response_create and session_update payloads, in the same GA shape, forwarded as the response.create and session.update client events. jambonz connects to wss://api.x.ai/v1/realtime.

1
session.llm({
2
  vendor: 'xai',
3
  model: 'grok-voice-latest',
4
  auth: { apiKey: process.env.XAI_API_KEY },
5
  actionHook: '/final',
6
  eventHook: '/event',
7
  toolHook: '/toolCall',
8
  llmOptions: {
9
    session_update: {
10
      type: 'realtime',
11
      instructions:
12
        'You are a friendly, helpful voice assistant on a phone call. ' +
13
        'Always respond in English unless the caller explicitly speaks another language. ' +
14
        'Keep responses concise and natural for spoken conversation.',
15
      turn_detection: { type: 'server_vad' },
16
      audio: {
17
        output: { voice: 'eve' }
18
      }
19
    },
20
    response_create: {
21
      output_modalities: ['audio'],
22
      instructions: 'Greet the caller warmly in English and ask how you can help today.'
23
    }
24
  }
25
});

Model selection

Voices

Set the voice via session_update.audio.output.voice (or response_create.audio.output.voice), same field as OpenAI. Available voices: eve (default), ara, rex, sal, leo.

Required session_update

Unlike OpenAI, llmOptions.session_update is required for xai — audio does not begin flowing until after the first session.updated server event is received.

Audio format

Audio on the wire to xAI is pcm16 at 24 kHz. jambonz forces this rate/format regardless of what is set in session_update.audio; do not attempt to select a different format or sample rate.

Turn detection

Unlike OpenAI GA, which nests turn detection under session_update.audio.input.turn_detection, xAI expects turn_detection top-level in session_update — i.e. session_update.turn_detection, not session_update.audio.input.turn_detection. { type: 'server_vad' } is the common setting. If turn_detection is omitted or set to null (manual turn mode), the application is responsible for turn-taking and can send input_audio_buffer.commit / input_audio_buffer.clear client events via an llm:update command.

Tool calls

Tool/function calling follows the same shape as OpenAI’s (session_update.tools, tool_choice). The completed tool call arrives on the response.function_call_arguments.done event rather than OpenAI’s response.output_item.done; jambonz routes it to your toolHook the same way.

Input transcription

When session_update.audio.input.transcription is configured, caller-speech transcripts arrive on conversation.item.input_audio_transcription.updated. Unlike OpenAI’s .completed event, this event is cumulative — each update contains the full transcript so far, not just a delta.

Google Gemini Live

Set vendor: 'google' to use Gemini’s Live API. The same vendor: 'google' integration reaches the Live API through two access paths:

Both speak the same JSON wire protocol — setup, realtimeInput, serverContent, modelTurn, toolCall, sessionResumptionUpdate are identical. Only the URL, auth, and model resource format differ. llmOptions.setup is forwarded verbatim to Google’s BidiGenerateContentSetup message after the WebSocket connects.

Gemini Developer API (API key)

Default access path. Provide an API key via auth.apiKey; the key starts with AIza. Model name uses the short form models/<id>. connectOptions is optional — defaults to the Gemini Developer endpoint.

1
session.llm({
2
  vendor: 'google',
3
  model: 'models/gemini-2.0-flash-live-001',
4
  auth: { apiKey: process.env.GEMINI_API_KEY },
5
  actionHook: '/final',
6
  eventHook: '/event',
7
  toolHook: '/toolCall',
8
  llmOptions: {
9
    setup: {
10
      generationConfig: {
11
        speechConfig: {
12
          voiceConfig: { prebuiltVoiceConfig: { voiceName: 'Aoede' } }
13
        }
14
      },
15
      systemInstruction: {
16
        parts: [{ text: 'You are a helpful assistant named Barbara.' }]
17
      }
18
    },
19
    greeting: 'Greet the caller warmly and ask how you can help.',
20
    sessionResumption: {}
21
  }
22
});

Vertex AI Live API (OAuth Bearer)

Production access path. The application mints an OAuth access token from a Google Cloud service-account JSON key and passes the token string (starting with ya29.) as auth.apiKey. jambonz detects the ya29. prefix and sends it as an Authorization: Bearer … header on the WebSocket upgrade. Model name must be the full Vertex resource path.

1
const { JWT } = require('google-auth-library');
2
const fs = require('fs');
3
4
const saKey = JSON.parse(fs.readFileSync(process.env.GOOGLE_SERVICE_ACCOUNT_KEY_PATH, 'utf8'));
5
const jwtClient = new JWT({
6
  email: saKey.client_email,
7
  key: saKey.private_key,
8
  scopes: ['https://www.googleapis.com/auth/cloud-platform']
9
});
10
11
const { token } = await jwtClient.getAccessToken();   // "ya29...."
12
const projectId = saKey.project_id;
13
const location = 'us-central1';
14
15
session.llm({
16
  vendor: 'google',
17
  model: `projects/${projectId}/locations/${location}/publishers/google/models/gemini-live-2.5-flash-native-audio`,
18
  auth: { apiKey: token },
19
  actionHook: '/final',
20
  eventHook: '/event',
21
  toolHook: '/toolCall',
22
  connectOptions: {
23
    host: `${location}-aiplatform.googleapis.com`,
24
    path: '/ws/google.cloud.aiplatform.v1beta1.LlmBidiService/BidiGenerateContent'
25
  },
26
  llmOptions: {
27
    setup: {
28
      generationConfig: {
29
        speechConfig: {
30
          voiceConfig: { prebuiltVoiceConfig: { voiceName: 'Aoede' } }
31
        }
32
      },
33
      systemInstruction: {
34
        parts: [{ text: 'You are a helpful assistant named Barbara.' }]
35
      }
36
    },
37
    greeting: 'Greet the caller warmly and ask how you can help.'
38
  }
39
});

Setup checklist for Vertex AI:

1. GCP project with aiplatform.googleapis.com API enabled and billing attached.
2. Service account with the roles/aiplatform.user IAM role on the project.
3. Service-account JSON key file accessible to the application (store outside repo, reference via env var).
4. Host region prefix in connectOptions.host must match the locations/<region> segment in model. e.g. us-central1-aiplatform.googleapis.com with projects/.../locations/us-central1/.... Mismatched regions return Publisher Model not found.

OAuth token lifetime: tokens issued from a service-account JWT expire after ~1 hour. jambonz uses whatever token the application provides at session start; it is not refreshed mid-session. The WebSocket stays open across expiry, but a reconnect with a stale token will fail. For typical call durations this is fine; mint a fresh token per call.

Google-specific connectOptions fields

Google-specific llmOptions fields

AssemblyAI Voice Agent

Set vendor: 'assemblyai' and supply your AssemblyAI API key via auth.api_key. llmOptions is the AssemblyAI Voice Agent session payload passed through verbatim — there is no jambonz-specific wrapper. jambonz wraps it as {type: 'session.update', session: <llmOptions>} and sends it as the first client message after the websocket connects to wss://agents.assemblyai.com/v1/ws. See the AssemblyAI Voice Agent product page and Voice Agent API docs for an overview.

The audio format is not configurable. AssemblyAI Voice Agent only accepts audio/pcm at 24 kHz, which jambonz uses unconditionally — session.input.format / session.output.format set by the application are overridden. jambonz resamples to/from the channel’s native rate automatically.

1
session.llm({
2
  vendor: 'assemblyai',
3
  auth: { api_key: process.env.ASSEMBLYAI_API_KEY },
4
  actionHook: '/final',
5
  eventHook: '/event',
6
  toolHook: '/toolCall',
7
  events: ['all'],
8
  llmOptions: {
9
    system_prompt: 'You are a helpful voice agent.',
10
    greeting: 'Hello, how can I help you today?',
11
    output: { voice: 'ivy' },
12
    input: {
13
      keyterms: ['weather', 'temperature'],
14
      turn_detection: {
15
        vad_threshold: 0.5,
16
        min_silence: 1000,
17
        max_silence: 3000,
18
        interrupt_response: true
19
      }
20
    },
21
    tools: [
22
      {
23
        type: 'function',
24
        name: 'getWeather',
25
        description: 'Get current weather for a given city',
26
        parameters: {
27
          type: 'object',
28
          properties: {
29
            location: { type: 'string', description: 'City name' },
30
            scale: { type: 'string', enum: ['celsius', 'fahrenheit'] }
31
          },
32
          required: ['location']
33
        }
34
      }
35
    ]
36
  }
37
});

AssemblyAI-specific auth fields

AssemblyAI-specific llmOptions fields

AssemblyAI’s protocol requires a session.update message, but every field inside is optional — pass llmOptions: {} to start with all server defaults.

Tool calls

The agent invokes a tool by emitting a tool.call server event. jambonz routes it to the application’s toolHook with {name, args, tool_call_id}. The application replies via session.sendToolOutput(tool_call_id, {type: 'tool.result', tool_call_id, result}). The result should be a string (JSON-stringify objects before sending) — jambonz JSON-stringifies non-string result values automatically.

Transfer-to-human handoff

Add a handoff block to let the realtime model transfer the caller to a human. The runtime injects a transfer_to_human tool — no toolHook is needed for it. When the caller asks for a human and the model calls the tool, jambonz runs the packaged transfer choreography to the configured destination.

1
session
2
  .llm({
3
    vendor: 'openai',
4
    model: 'gpt-realtime',
5
    auth: { apiKey: process.env.OPENAI_API_KEY },
6
    llmOptions: {
7
      response_create: { instructions: 'You are a helpful support agent.' },
8
      session_update: { type: 'realtime', instructions: 'You are a helpful support agent.' },
9
    },
10
    handoff: {
11
      mode: 'blind',
12
      blindMethod: 'dial',
13
      target: [{ type: 'user', name: 'agent-desk@sip.example.com' }],
14
    },
15
    actionHook: '/llm-complete',
16
  })
17
  .send();

The handoff block accepts every transfer option (mode, target, blindMethod, disposition, confirm, etc.), plus brief ('auto' | 'none' | { template }), briefSynthesizer, toolName, and toolDescription — see the agent verb’s handoff section for the details, which apply identically here. When the human leg bridges, the actionHook reports the transfer outcome.

Built-in Hangup Tool

Set hangup to let the model end the call on its own. The runtime injects a hangup tool into the model’s toolset; you do not define it in llmOptions and you do not handle it in your toolHook — the runtime intercepts the call, hangs up, and ends the llm verb. This works the same way across every supported s2s vendor (OpenAI, Deepgram Voice Agent, Ultravox, ElevenLabs, Google Gemini Live, AssemblyAI, xAI Voice Agent, OpenAI GPT Live — the last requires delegation.type: 'responses').

1
session.llm({
2
  vendor: 'openai',
3
  model: 'gpt-realtime',
4
  auth: { apiKey },
5
  hangup: { reason: 'conversation complete' },
6
  actionHook: '/final',
7
  llmOptions: {
8
    session_update: {
9
      type: 'realtime',
10
      instructions:
11
        'You are a helpful voice assistant. When the caller says goodbye, call the hangup tool.',
12
    },
13
  },
14
});

The injected tool accepts an optional reason argument that the model may fill in when it decides to end the call. The reason placed in the X-Reason SIP header on the outbound BYE is resolved as follows:

1. The model-supplied reason argument, if present.
2. Otherwise the app-supplied hangup.reason default, if configured.
3. Otherwise no X-Reason header is sent.

Pass an empty object (hangup: {}) to enable the tool with no default reason.

Example Applications

Please checkout the following example applications:

• for OpenAI
• for Deepgram
• for Ultravox
• for ElevenLabs
• for Google Gemini (Developer API + Vertex AI)
• for AssemblyAI
• for OpenAI GPT Live