Streaming Text-to-Speech
Stream synthesized speech to a callback as it is generated, so you can restream it to a customer. Works with Cartesia, ElevenLabs, OpenAI, Mistral and Gemini.
aiSpeakStream() sends text to a provider and calls your function with audio as it arrives, instead of waiting for the whole file. Use it to restream audio to a caller, a browser, or a telephony bridge.
aiSpeakStream(
"Thanks for calling BoxLang support.",
( event ) => {
if ( event.type == "audio" ) {
socket.send( event.data ) // binary audio, ready to forward
}
},
{ voice: "female" },
{ provider: "cartesia", outputFormat: "pcm" }
)π§ Syntax
summary = aiSpeakStream( text, callback, params={}, options={} )
// Fluent
summary = aiSpeak().text( "Hello" ).provider( "cartesia" ).asPCM().stream( callback )text
string
The text to synthesize. Required and non-empty
callback
function
Called with one event struct per event (see below)
params
struct
Provider parameters: model, voice, sample_rate, language, and so on
options
struct
provider, apiKey, outputFormat, timeout (idle timeout in seconds), logging. outputFile is ignored when streaming
π¨ Callback events
Every event is a struct with a type:
type
Fields
When
audio
data (binary), format, sampleRate, sequence
Each chunk of audio. sampleRate is 0 when the provider does not report one
timestamps
words, start, end
Cartesia over SSE when you pass add_timestamps: true
done
chunks, bytes
Always the last event, even when you stop early
Stop early. Return an explicit false from the callback to stop (for example when the caller interrupts the agent). Any other return value, including nothing, keeps streaming. The done event is still sent and the summary reports completed: false.
Stopping closes the connection to the provider immediately, for both raw-byte and SSE streams, so generation stops and the final done event arrives right away.
β²οΈ Timeout
options.timeout (default 30 seconds) is an idle timeout: the longest the stream will wait for the response headers or between received bytes, on both success and error responses. A provider that stalls is aborted with a ProviderError, while a healthy stream of any length is never cut off.
π Return value
A summary struct:
provider, model
Who produced the audio
audioFormat, sampleRate
The format label placed on each chunk
chunks, bytes
Audio chunks and total bytes delivered
completed
false if your callback stopped the stream
π§© Provider support
Cartesia
β
/tts/bytes for mp3 and wav, /tts/sse for raw formats
mp3, wav, pcm, mulaw, alaw
SSE adds word timestamps. Force a transport with params.transport (bytes or sse). mp3 and wav cannot use SSE
ElevenLabs
β
/stream chunked bytes
mp3, pcm, mulaw, alaw, opus
wav cannot stream. Stream pcm instead
OpenAI
β
Chunked bytes
Any response_format. pcm (24kHz) and wav start fastest
params.stream_format is ignored
Mistral
β
SSE (stream: true)
mp3, wav, pcm, flac, opus
voice is sent as voice_id: a UUID as is, a voice name after a lookup
Gemini
β
Interactions API SSE
Raw 24kHz 16-bit mono PCM, labelled pcm
Headerless PCM. Wrap it yourself for a container
Grok / xAI
β
WebSocket only (not implemented)
Throws UnsupportedCapability. Use aiSpeak()
Others
β
Throw UnsupportedCapability
A provider can stream when service.hasCapability( "speechStream" ) is true.
π Formats for voice agents
Browser playback
mp3
Widely supported. Cartesia and ElevenLabs stream it
Voice agent pipeline
pcm
Raw, lowest latency, no decoding. Choose the rate with params.sample_rate (Cartesia default 24000)
Telephony (Twilio, SIP)
mulaw or alaw
8kHz companded audio is what phone networks expect
β±οΈ Word timestamps (Cartesia)
β οΈ Errors
UnsupportedCapability
The provider cannot stream speech
InvalidArgument
Empty text, an unsupported format for the provider, or an invalid transport
ProviderError
The provider rejected the request. The message includes the provider's explanation
π Events
aiSpeakStream() fires beforeAISpeech and afterAISpeech (with stream: true in the event data), plus onAISpeakRequest from the provider.
π§ Try it
Working demos that play streamed speech in a browser:
bx-ai
examples/http-streaming-speech:<audio>mode (mp3) and voice-agent mode (PCM with word highlighting and barge-in)bx-ai-intro
examples/speech-stream-http: same demo with the intro setupbx-ai-intro examples 41 and 42: CLI walkthroughs of streaming and Cartesia
See Also
Last updated