> For clean Markdown of any page, append .md to the page URL.
> For a complete documentation index, see https://docs.jambonz.org/llms.txt.
> For AI client integration (Claude Code, Cursor, etc.), connect to the MCP server at https://docs.jambonz.org/_mcp/server.

# Using OpenAI STT

jambonz supports a wide range of speech recognition vendors, and when we add support for new speech
vendor we try to support and expose all of their options so that you can fully utilize their capabilities.

OpenAI is rather unique in that it supports a prompt feature that allows you to pass in a custom prompt
to help guide the recognizer.

> **Info**
>
> This is something [we have been asking STT vendors](https://blog.jambonz.org/speech-companies-are-failing-at-conversational-ai#heading-api-interface-that-includes-relevant-prompt) for a while.

In this article we explore the different ways to exploit the prompt feature of OpenAI STT.

To begin with, here are the possible options that you use with OpenAI STT:

```
  recognizer: {
    vendor: 'openai',
    ..other recognition options
    openaiOptions: {
      model: 'gpt-4o-transcribe', // or 'gpt-4o-mini-transcribe', 'whisper-1',
                                  // 'gpt-live-transcribe', or 'gpt-realtime-whisper'
      input_audio_noise_reduction: 'near_field', // or 'far_field'
      prompt: 'string',
      turn_detection: {
        type: 'server_vad', // or 'semantic_vad' or 'none'
        eagerness: 'medium', // only for semantic_vad: 'low', 'medium', 'high', or 'auto'
        prefix_padding_ms: 300, // only for server_vad
        silence_duration_ms: 800 // only for server_vad
      },
      // only for 'gpt-live-transcribe'
      languages: ['en', 'fr'], // language hints as a LIST; do not combine with language
      keywords: ['jambonz', 'drachtio'], // literal term hints; defaults to the recognizer hints
      delay: 'low', // 'minimal', 'low', 'medium', 'high', or 'xhigh' — latency/accuracy trade-off
      // local VAD knobs, only used with the client-endpointed models
      // ('gpt-live-transcribe' and 'gpt-realtime-whisper')
      vadMode: 2,        // 0-3, higher is more aggressive
      vadSilenceMs: 500, // ms of silence to end an utterance
      vadVoiceMs: 250,   // ms of voice to start an utterance
      promptTemplates: {
        hintsTemplate: 'string',
        conversationHistoryTemplate: 'string',
      }
    }
  }
```

### Turn detection options

The `turn_detection` object controls how OpenAI detects when a speaker has finished talking.

**For `semantic_vad` type:**

* `eagerness`: Controls how eager the model is to determine the end of an utterance. Possible values:
  * `auto` (default): Equivalent to `medium`
  * `low`: Allows the user more time to speak, resulting in larger transcript chunks
  * `medium`: Balanced approach
  * `high`: Returns transcription events faster with smaller chunks

The `eagerness` setting affects how audio is chunked even in transcription mode. Use `high` if you want faster transcription events, or `low` if you prefer larger, more complete transcript chunks.

### Client-endpointed models: `gpt-live-transcribe` and `gpt-realtime-whisper`

These two models do not support OpenAI's server-side turn detection — the API rejects a
`turn_detection` block for them, and left alone they would stream partial text forever without
ever finalizing a transcript. jambonz therefore detects the end of each utterance itself, using
a local voice-activity detector on the media server, and tells OpenAI when to finalize.

Any `turn_detection` settings in `openaiOptions` are ignored when these models are selected.
Instead, the local VAD knobs apply:

* `vadMode` (0–3, default 2): detection aggressiveness — higher opens a turn on quieter speech
* `vadVoiceMs` (default 250): milliseconds of sustained voice that starts an utterance
* `vadSilenceMs` (default 500): milliseconds of silence that ends an utterance and finalizes it

Partial deltas continue to be delivered while the user is speaking (when `interim` is set on the
recognizer), and a single final transcript is delivered per utterance once the local VAD detects
silence.

```
{
  recognizer: {
    vendor: 'openai',
    openaiOptions: {
      model: 'gpt-realtime-whisper',
    }
  }
}
```

### Using `gpt-live-transcribe`

`gpt-live-transcribe` is OpenAI's newest realtime transcription model. In addition to the
client-endpointed behavior above, it accepts three fields the other models do not:

* `languages`: language hints as a **list** (e.g. `['en', 'fr']`). This replaces the singular
  `language` for this model — jambonz sends whichever form the model accepts, so set one or the
  other, not both. When unset, the recognizer's `language` is used.
* `keywords`: literal term hints — product names, acronyms, medications — that the model should
  be primed to recognize. These are hints, not required output. When unset, the recognizer's
  `hints` are used.
* `delay`: the latency/accuracy trade-off, one of `minimal`, `low`, `medium`, `high`, or
  `xhigh`. Lower emits partial text sooner; higher gives the model more audio context per chunk
  and can improve accuracy.

```
{
  recognizer: {
    vendor: 'openai',
    interim: true,
    openaiOptions: {
      model: 'gpt-live-transcribe',
      keywords: ['jambonz', 'drachtio', 'SIP'],
      delay: 'low',
      vadSilenceMs: 500,
    }
  }
}
```

In this article we want to explore the various ways to construct a prompt for OpenAI STT.

## Providing hints

To start with the simplest method, if you provide hints and you are using 'whisper-1' as the model,
then the hints will simply be used as the prompt.

```
{
  recognizer: {
    vendor: 'openai',
    hints: ['DaLL-E', 'GPT-4', 'ChatGPT', 'jambonz'],
    openaiOptions: {
      model: 'whisper-1',
    }
  }
}

// prompt => DaLL-E, GPT-4, ChatGPT, jambonz
```

The reason for this is that the 'whisper-1' model supports a limited number of tokens in the prompt so it
is recommended to simply use the hints as the prompt.  Note that this is the default behavior, but if you
specify either 'prompt' or 'promptTemplates' then the prompt will be generated from those settings and not the hints.

## Using the prompt setting

You can also simply provide the prompt using the prompt setting.

```
{
    recognizer: {
    vendor: 'openai',
    openaiOptions: {
      model: 'gpt-4o-transcribe',
      prompt: 'The user is providing a credit card number',
    }
  }
}

// prompt => The user is providing a credit card number
```

## Using promptTemplates

A further option is to use the promptTemplates options.  These give you the ability to provide a template that
is interpolated with either the hints or the conversation history to create a final prompt.

```
{
  recognizer: {
    vendor: 'openai',
    hints: ['DaLL-E', 'GPT-4', 'ChatGPT', 'jambonz'],
    openaiOptions: {
      model: 'gpt-4o-transcribe',
      promptTemplates: {
        hintsTemplate: 'Please spell the following words properly: {{hints}}',
      }
    }
  }
}

// prompt => Please spell the following words properly: DaLL-E, GPT-4, ChatGPT, jambonz
```

or, using conversation history:

```
{
  recognizer: {
    vendor: 'openai',
    openaiOptions: {
      model: 'gpt-4o-transcribe',
      promptTemplates: {
         conversationHistoryTemplate: 'Here is the recent conversation history: {{turns}}',
      }
    }
  }
}

// prompt => Here is the recent conversation history:
// assistant: Hello, how can I help you today?
// user: My internet is broken.
// assistant: Could you please tell me your address?
// user:
```

By default, the conversation history is limited to the last 4 turns you can adjust this as well.

```
{
  recognizer: {
    vendor: 'openai',
    openaiOptions: {
      model: 'gpt-4o-transcribe',
      promptTemplates: {
         conversationHistoryTemplate: 'Here is the recent conversation history: {{turns:1}}',
      }
    }
  }
}

// prompt => Here is the recent conversation history:
// assistant: Could you please tell me your address?
// user:
```

Note that you can provide both hintsTemplate and conversationHistoryTemplate and the final prompt will
concatenate the two interpolated strings.