Using OpenAI STT

jambonz supports a wide range of speech recognition vendors, and when we add support for new speech vendor we try to support and expose all of their options so that you can fully utilize their capabilities.

OpenAI is rather unique in that it supports a prompt feature that allows you to pass in a custom prompt to help guide the recognizer.

This is something we have been asking STT vendors for a while.

In this article we explore the different ways to exploit the prompt feature of OpenAI STT.

To begin with, here are the possible options that you use with OpenAI STT:

  recognizer: {
    vendor: 'openai',
    ..other recognition options
    openaiOptions: {
      model: 'gpt-4o-transcribe', // or 'gpt-4o-mini-transcribe', 'whisper-1',
                                  // 'gpt-live-transcribe', or 'gpt-realtime-whisper'
      input_audio_noise_reduction: 'near_field', // or 'far_field'
      prompt: 'string',
      turn_detection: {
        type: 'server_vad', // or 'semantic_vad' or 'none'
        eagerness: 'medium', // only for semantic_vad: 'low', 'medium', 'high', or 'auto'
        prefix_padding_ms: 300, // only for server_vad
        silence_duration_ms: 800 // only for server_vad
      },
      // only for 'gpt-live-transcribe'
      languages: ['en', 'fr'], // language hints as a LIST; do not combine with language
      keywords: ['jambonz', 'drachtio'], // literal term hints; defaults to the recognizer hints
      delay: 'low', // 'minimal', 'low', 'medium', 'high', or 'xhigh' — latency/accuracy trade-off
      // local VAD knobs, only used with the client-endpointed models
      // ('gpt-live-transcribe' and 'gpt-realtime-whisper')
      vadMode: 2,        // 0-3, higher is more aggressive
      vadSilenceMs: 500, // ms of silence to end an utterance
      vadVoiceMs: 250,   // ms of voice to start an utterance
      promptTemplates: {
        hintsTemplate: 'string',
        conversationHistoryTemplate: 'string',
      }
    }
  }

Turn detection options

The turn_detection object controls how OpenAI detects when a speaker has finished talking.

For semantic_vad type:

eagerness: Controls how eager the model is to determine the end of an utterance. Possible values:
- auto (default): Equivalent to medium
- low: Allows the user more time to speak, resulting in larger transcript chunks
- medium: Balanced approach
- high: Returns transcription events faster with smaller chunks

The eagerness setting affects how audio is chunked even in transcription mode. Use high if you want faster transcription events, or low if you prefer larger, more complete transcript chunks.

Client-endpointed models: `gpt-live-transcribe` and `gpt-realtime-whisper`

These two models do not support OpenAI’s server-side turn detection — the API rejects a turn_detection block for them, and left alone they would stream partial text forever without ever finalizing a transcript. jambonz therefore detects the end of each utterance itself, using a local voice-activity detector on the media server, and tells OpenAI when to finalize.

Any turn_detection settings in openaiOptions are ignored when these models are selected. Instead, the local VAD knobs apply:

vadMode (0–3, default 2): detection aggressiveness — higher opens a turn on quieter speech
vadVoiceMs (default 250): milliseconds of sustained voice that starts an utterance
vadSilenceMs (default 500): milliseconds of silence that ends an utterance and finalizes it

Partial deltas continue to be delivered while the user is speaking (when interim is set on the recognizer), and a single final transcript is delivered per utterance once the local VAD detects silence.

{
  recognizer: {
    vendor: 'openai',
    openaiOptions: {
      model: 'gpt-realtime-whisper',
    }
  }
}

Using `gpt-live-transcribe`

gpt-live-transcribe is OpenAI’s newest realtime transcription model. In addition to the client-endpointed behavior above, it accepts three fields the other models do not:

languages: language hints as a list (e.g. ['en', 'fr']). This replaces the singular language for this model — jambonz sends whichever form the model accepts, so set one or the other, not both. When unset, the recognizer’s language is used.
keywords: literal term hints — product names, acronyms, medications — that the model should be primed to recognize. These are hints, not required output. When unset, the recognizer’s hints are used.
delay: the latency/accuracy trade-off, one of minimal, low, medium, high, or xhigh. Lower emits partial text sooner; higher gives the model more audio context per chunk and can improve accuracy.

{
  recognizer: {
    vendor: 'openai',
    interim: true,
    openaiOptions: {
      model: 'gpt-live-transcribe',
      keywords: ['jambonz', 'drachtio', 'SIP'],
      delay: 'low',
      vadSilenceMs: 500,
    }
  }
}

In this article we want to explore the various ways to construct a prompt for OpenAI STT.

Providing hints

To start with the simplest method, if you provide hints and you are using ‘whisper-1’ as the model, then the hints will simply be used as the prompt.

{
  recognizer: {
    vendor: 'openai',
    hints: ['DaLL-E', 'GPT-4', 'ChatGPT', 'jambonz'],
    openaiOptions: {
      model: 'whisper-1',
    }
  }
}
// prompt => DaLL-E, GPT-4, ChatGPT, jambonz

The reason for this is that the ‘whisper-1’ model supports a limited number of tokens in the prompt so it is recommended to simply use the hints as the prompt. Note that this is the default behavior, but if you specify either ‘prompt’ or ‘promptTemplates’ then the prompt will be generated from those settings and not the hints.

Using the prompt setting

You can also simply provide the prompt using the prompt setting.

{
    recognizer: {
    vendor: 'openai',
    openaiOptions: {
      model: 'gpt-4o-transcribe',
      prompt: 'The user is providing a credit card number',
    }
  }
}
// prompt => The user is providing a credit card number

Using promptTemplates

A further option is to use the promptTemplates options. These give you the ability to provide a template that is interpolated with either the hints or the conversation history to create a final prompt.

{
  recognizer: {
    vendor: 'openai',
    hints: ['DaLL-E', 'GPT-4', 'ChatGPT', 'jambonz'],
    openaiOptions: {
      model: 'gpt-4o-transcribe',
      promptTemplates: {
        hintsTemplate: 'Please spell the following words properly: {{hints}}',
      }
    }
  }
}
// prompt => Please spell the following words properly: DaLL-E, GPT-4, ChatGPT, jambonz

or, using conversation history:

{
  recognizer: {
    vendor: 'openai',
    openaiOptions: {
      model: 'gpt-4o-transcribe',
      promptTemplates: {
         conversationHistoryTemplate: 'Here is the recent conversation history: {{turns}}',
      }
    }
  }
}
// prompt => Here is the recent conversation history:
// assistant: Hello, how can I help you today?
// user: My internet is broken.
// assistant: Could you please tell me your address?
// user:

By default, the conversation history is limited to the last 4 turns you can adjust this as well.

{
  recognizer: {
    vendor: 'openai',
    openaiOptions: {
      model: 'gpt-4o-transcribe',
      promptTemplates: {
         conversationHistoryTemplate: 'Here is the recent conversation history: {{turns:1}}',
      }
    }
  }
}
// prompt => Here is the recent conversation history:
// assistant: Could you please tell me your address?
// user:

Note that you can provide both hintsTemplate and conversationHistoryTemplate and the final prompt will concatenate the two interpolated strings.