OpenAI GPT Live

This is an example jambonz application that connects a phone call to OpenAI’s GPT Live API. It answers the call, asks the agent to greet the caller, and implements a “get weather” function the agent can call.

Running the example

$
git clone https://github.com/jambonz/v10-examples.git
$
cd v10-examples/examples/s2s/gptlive
$
npm install
$
npm start

The app listens on ws://localhost:3000/ by default (PORT and LOG_LEVEL are ordinary environment variables). Create a jambonz application pointing its calling webhook at that URL over WebSocket, and assign a phone number to it.

Configuration

Everything else is configured with application variables, which you set in the jambonz portal — the example declares them so the portal discovers them automatically, and reads them from session.data.env_vars at call time. They are not shell environment variables.

Configuring the assistant

All the interesting code is in src/index.ts. Configuration goes in a session_update inside llmOptions:

1
session.s2s({
2
  vendor: 'gptlive',
3
  model: env.GPTLIVE_MODEL,
4
  auth: { apiKey: env.GPTLIVE_API_KEY },
5
  llmOptions: {
6
    session_update: {
7
      instructions: 'You are a friendly and helpful voice assistant for Jambonz Mobile. '
8
        + 'Keep your responses concise and conversational. '
9
        + 'You are speaking via voice, so respond in plain prose with no markdown.',
10
      audio: {
11
        output: { voice: 'marin' },
12
      },
13
      delegation,   // see below
14
    },
15
  },
16
  toolHook: '/tool-call',
17
  eventHook: '/s2s-event',
18
  actionHook: '/s2s-complete',
19
});

voice accepts the GPT Live voice names; the example defaults to marin.

Two things to note if you are used to the other speech-to-speech tutorials:

• There is no response_create. GPT Live has no such client event.
• There is nothing to configure for audio format or turn detection. GPT Live fixes the audio at 24 kHz mono PCM and handles turn detection itself, and jambonz converts to and from the caller’s codec for you.

session_update is required — GPT Live will not accept the caller’s audio until it has your configuration.

Delegations

This is the part of GPT Live with no equivalent in the other vendors. Whenever the model needs something from outside the spoken conversation, it creates a delegation. You choose which kind up front, and the choice decides whether you can use function calling at all.

The example exposes this as the DELEGATION_MODE application variable so you can try both.

responses — the agent can call your functions

1
const delegation = {
2
  type: 'responses',
3
  responses: {
4
    model: 'gpt-5.5',
5
    tools: [weatherTool],
6
  },
7
};

The nested responses object is required, and so is its model — that is a second model, which runs the delegated turn, and it is separate from the voice model on the verb. Your tool definitions go in responses.tools, not at the top level of the session.

Use this mode if you want function calling, MCP servers, or jambonz’s built-in handoff and hangup tools.

client — the agent asks your app for context

1
const delegation = { type: 'client' };

In this mode the model asks your application for background information in prose rather than calling a function. You get a delegation.created event and answer it with up to 500 tokens of text:

1
session.on('/s2s-event', (evt) => {
2
  if (evt.type === 'delegation.created' && evt.item?.target === 'client') {
3
    session.updateLlm({
4
      type: 'delegation.context.append',
5
      delegation_item_id: evt.item.id,
6
      content: [{
7
        type: 'input_text',
8
        text: 'The caller is a Jambonz Mobile customer on the Unlimited plan. '
9
          + 'Their account is in good standing.',
10
      }],
11
    });
12
  }
13
});

Greeting the caller

Because there is no response_create, nothing tells the model to take the first turn — and putting the greeting in instructions does not work. The model waits for the caller, who hears silence.

To open the call, ask for the greeting when you receive session.started:

1
session.on('/s2s-event', (evt) => {
2
  if (evt.type === 'session.started') {
3
    session.updateLlm({
4
      type: 'session.context.append',
5
      content: [{
6
        type: 'input_text',
7
        text: 'Immediately greet the caller using the exact text below. Do not wait for the '
8
          + 'caller to speak first. After the greeting, pause and listen.\n\n'
9
          + 'Hi, I am the Jambonz Mobile assistant. How can I help you today?',
10
      }],
11
    });
12
  }
13
});

Give it both the wording you want and an instruction about when to speak. Leave the wording out and the agent will use its own.

Function calling

The example implements a get_weather function using the free APIs from open-meteo.com. The tool is declared in delegation.responses.tools, and jambonz calls your toolHook when the model wants to run it — the same as every other vendor.

What differs is the envelope you return the result in:

1
session.on('/tool-call', async (evt) => {
2
  const { tool_call_id, name, args } = evt;
3
  const result = await lookupWeather(args.location);
4
5
  session.sendToolOutput(tool_call_id, {
6
    type: 'delegation.function_call_output.create',
7
    item: {
8
      type: 'function_call_output',
9
      call_id: tool_call_id,
10
      output: result,          // must be a string
11
    },
12
  });
13
});

Unlike the Realtime API, there is no follow-on response.create to send — the server picks the conversation back up on its own once it has your result.

Events

Name the events you want in the events property of the verb — if you omit it, jambonz forwards everything, including high-volume transcript fragments. The reference documentation has the full list; GPT Live is in alpha and has no public event reference of its own.

The most useful one for following the conversation is turn.done, which carries a completed utterance and a turn.role of 'user' or 'assistant':

1
session.on('/s2s-event', (evt) => {
2
  if (evt.type === 'turn.done') {
3
    log.info({ role: evt.turn?.role, transcript: evt.turn?.transcript }, 'turn');
4
  }
5
});

If you want live partials instead, input_transcript.added and output_transcript.added stream fragments as speech is recognized — but their boundaries follow speech cadence rather than complete thoughts, so one sentence may arrive in several pieces.

Barge-in needs no work on your part: when the caller talks over the agent, jambonz flushes the queued audio automatically.

actionHook properties

Like many jambonz verbs, the llm verb sends an actionHook with a final status when the session completes. Handle it and acknowledge it — on WebSocket transport a session that never replies will hang:

1
session.on('/s2s-complete', (evt) => {
2
  log.info(evt, 's2s complete');
3
  session.reply();
4
});

The payload includes a completion_reason explaining why the session ended. The reference documentation lists the values in full; the one you are most likely to see while getting started is server error, which almost always means the API key is not enrolled in the Early Access Program. The payload carries an error object with OpenAI’s own reason — check that first.

Note that not every problem ends the call. Once the session is running, a rejected client event or a failed delegation is reported on your eventHook and the conversation carries on, so you can recover if you want to.

Resources

• The llm verb reference — full field and event documentation
• Migrating from the OpenAI Realtime API
• The say verb — for greetings that must use exact wording
• The example application