Qwen Omni Realtime

This is an example jambonz application that connects to Alibaba’s Qwen Omni-Realtime API (the models behind Qwen-Audio-3.0-Realtime: qwen3.5-omni-plus-realtime and qwen3.5-omni-flash-realtime) and illustrates how to build a Voice-AI application using jambonz and Qwen. The wire protocol is an OpenAI-Realtime dialect, so if you have used the OpenAI Realtime API with jambonz this will feel very familiar.

Authentication

You must have an Alibaba Cloud Model Studio (DashScope) API key with access to the Omni-Realtime models. Provide it in the auth property of the llm verb (or as an application environment variable, as the example does).

1
session.qwen_s2s({
2
  model: 'qwen3.5-omni-flash-realtime',
3
  auth: {
4
    apiKey: process.env.DASHSCOPE_API_KEY,
5
    // host is optional; the default is the international endpoint
6
    // 'dashscope-intl.aliyuncs.com'. Set a workspace-scoped host
7
    // ('ws-<workspaceId>.<region>.maas.aliyuncs.com', recommended by Alibaba)
8
    // or 'dashscope.aliyuncs.com' for the China (Beijing) region.
9
  },
10
  ...
11
})

Configuring the assistant

Configuration is provided in the form of session_update and (optionally) response_create client events in the llmOptions property. session_update is required — audio is gated until Qwen acknowledges the session configuration — and carries the voice (55 voices; the example uses Ethan), system instructions, and turn detection:

1
llmOptions: {
2
  session_update: {
3
    modalities: ['text', 'audio'],
4
    voice: 'Ethan',
5
    instructions: 'You are a friendly and helpful voice assistant.',
6
    turn_detection: {
7
      type: 'semantic_vad',
8
      threshold: 0.5,
9
      silence_duration_ms: 800,
10
    },
11
  },
12
}

Turn detection

Qwen offers two server-side turn detection modes: server_vad (acoustic only — the server default) and semantic_vad, which weighs whether the caller’s utterance sounds complete before ending the turn and is the mode we recommend starting with. The silence_duration_ms value controls how long a pause ends the caller’s turn: lower values respond faster but will interrupt callers who pause mid-sentence; 800 ms (the vendor default) is a reasonable starting point.

Greeting the caller

By default the assistant stays silent until the caller speaks. To have the assistant speak first, include a response_create in llmOptions — jambonz sends it immediately after the session is configured:

1
llmOptions: {
2
  session_update: { ... },
3
  response_create: {
4
    instructions: 'Greet the caller warmly and ask how you can help them today.',
5
  },
6
}

Interrupting the assistant

Qwen handles barge-in server-side (interrupt_response is enabled by default in turn_detection): when the caller speaks over the assistant, the server stops generating and jambonz flushes any queued assistant audio the moment the input_audio_buffer.speech_started event arrives. Note that barge-in responsiveness is governed by the same turn_detection configuration as end-of-turn: server_vad interrupts faster and more consistently, while semantic_vad may deliberate briefly before signaling speech.

Events

Qwen sends OpenAI-Realtime-style server events (session.updated, input_audio_buffer.speech_started/speech_stopped, streaming caller transcription via conversation.item.input_audio_transcription.delta/completed, assistant transcripts via response.audio_transcript.delta/done, response.done, and so on). Your application specifies which it wants to receive in the events property of the llm verb, with wildcard support (e.g. "conversation.item.*"). The response.audio.delta event contains the actual audio and is consumed by jambonz itself.

actionHook properties

Like many jambonz verbs, the llm verb sends an actionHook with a final status when the verb completes. The payload includes a completion_reason property indicating why the session completed, one of:

• normal conversation end
• connection failure
• disconnect from remote end
• server failure
• server error

Limits worth knowing

Qwen sessions are capped at 120 minutes, and conversation history within a session is capped per model tier (for example, 80 audio turns on qwen3.5-omni-flash-realtime). Function calling is supported, but Qwen’s built-in web search and tool calling are mutually exclusive within a single session.