TTS Services

The text-to-speech layer turns assistant text into streamed audio. It ships as two independent component services that speak the same WebSocket envelope protocol, plus a fallback owner in the conversation engine that chains them together.

Two backends

Both services expose the same synthesis contract and stream base64-encoded PCM16LE, mono, 24000 Hz audio. They differ only in how audio is produced and how a per-avatar voice is selected.

Backend	Role	Port	Module	Voice mechanism
tts_service (ElevenLabs)	Primary	8014 (TTS_ELEVENLABS_PORT)	component_services.tts_service.main:app	Cloud ElevenLabs; per-avatar elevenlabs_voice_id
tts_omnivoice (OmniVoice)	Fallback	8015 (TTS_OMNIVOICE_PORT)	component_services.tts_omnivoice:app	Local GPU model k2-fsa/OmniVoice; reference-audio voice cloning

Both bind 127.0.0.1 and are launched via run/run.sh / run/services/start.sh with uvicorn.

Neither service knows about the other. The primary/fallback relationship is owned entirely by the conversation engine's FallbackTTSClient (conversation_engine/services/tts.py).

note

tts_service is a layered (hexagonal) ElevenLabs adapter: an inbound WS API (api/), an application core (application/) with a TTSProviderPort, and an outbound ElevenLabsClient (services/elevenlabs.py). tts_omnivoice is a single file (component_services/tts_omnivoice.py) containing the app, WS handler, model manager, voice-profile store, and metrics.

Fallback behavior

FallbackTTSClient.synthesize runs the ElevenLabs primary (ENGINE_CONFIG.tts_ws_url, 8014) and awaits the first AudioChunk from its generator. The fallback to OmniVoice (ENGINE_CONFIG.tts_omnivoice_ws_url, 8015) fires when that first chunk never arrives:

• StopAsyncIteration — the primary produced zero audio chunks (an upstream error event or a clean close broke the engine TTSClient loop before any audio), or
• any other Exception — WebSocket connect failure or open_timeout.

The same request kwargs (text, session_id, avatar_id, language_hint, voice_clone_path, voice_clone_text, elevenlabs_voice_id) are then replayed against the fallback backend.

warning

Fallback is first-chunk-only. Once the first primary chunk is yielded, the engine drains the rest of the primary stream and returns — a mid-stream primary failure is not retried against OmniVoice.

The engine's single-backend TTSClient swallows failures by ending its generator: a websockets.WebSocketException is logged and the stream ends; an upstream error event is logged and breaks the loop. Both surface to FallbackTTSClient as "no chunks" and thus trigger the fallback.

Inside the pipeline, TTS is a best-effort final step: it runs only when the session has voice_output enabled and there is assistant text, and the whole synthesis loop is wrapped in a try/except that swallows exceptions. If both backends fail, the turn still completes without audio.

The gateway relay path is different — see Voiceover flow below. It has no fallback.

Endpoints

The synthesis WebSocket lives at TTS_CONFIG.ws_path (default /ws/tts) on each service. These routes are not mounted behind the gateway /api_lis prefix; they are consumed directly by the conversation engine and by the gateway relay.

ElevenLabs (tts_service, :8014)

Method	Path	Description
WS	/ws/tts	Synthesis stream. Inbound synthesis.start / session.ping / session.cancel; outbound synthesis.started → audio.chunk* → synthesis.completed.
GET	/health	status ok/degraded based on api_key + voice_id + model_id; backend: "elevenlabs".
GET	/status	Config snapshot: sample_rate, audio_format, channels, max_text_chars, base_url, voice_id, model_id, output_format, api_key_set.

OmniVoice (tts_omnivoice, :8015)

Method	Path	Description
WS	/ws/tts	Same protocol; synthesis.completed also carries latency and a profiling block. Always-clone voice.
GET	/health	ready (model loaded and profile store ok), model_error, gpu_visible, voice_profile_store, active_requests, cache_size; backend: "omnivoice".
GET	/status	In-memory metrics plus config (model_id, device, dtype, num_step_default, pcm_stream_ms, language_default, max_concurrent_synthesis, …).

Gateway voiceover relay

The gateway exposes one public TTS path. It proxies a synthesis.start session directly to the ElevenLabs service and injects the avatar's voice_clone_path / voice_clone_text into the first frame (only if not already present).

Method	Path	Description
WS	/api_lis/voiceover	Relays to GATEWAY_CONFIG.tts_ws_url (ElevenLabs 8014 only). Query params user_id, api_key, session_id. No OmniVoice fallback. On upstream WS/OS error the client socket is closed with code 1014.

note

The ElevenLabs → OmniVoice fallback is exercised only on the conversation path (/api_lis/sessions/{session_id}/message with voice_output=true), where the engine pipeline drives FallbackTTSClient and delivers audio inline. See the WebSocket Protocol page for the voiceover and message flows.

Synthesis protocol

Accepted inbound events on /ws/tts (both backends): synthesis.start, session.ping, session.cancel.

tip

Note the asymmetry: the inbound trigger is synthesis.start, while the first outbound event is synthesis.started.

The outbound sequence:

synthesis.started
  → voice.resolved        (only when payload.debug = true)
  → audio.chunk           (repeated)
  → synthesis.completed

audio.chunk payload fields:

Field	Value
chunk_index	1-based chunk counter
audio	base64-encoded PCM
audio_format	pcm16le
sample_rate	24000
channels	1
is_final	false
duration_ms	chunk duration

Backend differences:

• ElevenLabs — synthesis.started reports request_state: "streaming_elevenlabs", backend: "elevenlabs".
• OmniVoice — synthesis.started reports request_state: "resolving_voice"; synthesis.completed adds latency_to_first_chunk_ms, voice_mode_used, reference_used, and a detailed profiling block.

Session control and concurrency

• The WebSocket binds to the first session_id it sees; a different session_id returns SESSION_CANCELLED.
• A second synthesis.start while a synthesis task is running returns REQUEST_ALREADY_IN_PROGRESS.
• session.cancel sets the stop event, cancels the running task, and replies session.cancelled. session.ping replies session.pong.
• ElevenLabs has no global concurrency limit (bounded only by one task per connection). OmniVoice throttles all connections through a BoundedSemaphore(TTS_MAX_CONCURRENT_SYNTHESIS) (default 1) plus a model lock — effectively one GPU synthesis at a time by default.

Request validation

Rule	Error code
stream: false	INVALID_PAYLOAD
audio_format ≠ pcm16le	INVALID_PAYLOAD
sample_rate ≠ 24000	INVALID_PAYLOAD
missing avatar_id or text	MISSING_REQUIRED_FIELD
empty text after markdown/URL sanitization	INVALID_PAYLOAD

Text is sanitized (markdown, URLs, citations stripped) and truncated to max_text_chars. MODEL_NOT_READY (OmniVoice model not loaded) and unhandled INTERNAL_ERROR are reported as non-recoverable (recoverable: false); most other errors are recoverable.

Voice cloning

Per-avatar voice selection is backend-specific. Both backends receive the same synthesis.start payload, but each ignores the other's fields.

ElevenLabs uses only elevenlabs_voice_id, validated against ^[A-Za-z0-9_-]{1,64}$ (otherwise INVALID_PAYLOAD); it falls back to ELEVENLABS_CONFIG.voice_id when absent. ElevenLabsClient.stream_pcm accepts reference_audio_path / reference_text to satisfy the port contract but ignores them — reference-audio cloning is not implemented for ElevenLabs.

OmniVoice is always-clone: resolve_voice always sets reference_used = true and voice_mode_used = "reference". Reference resolution precedence:

1. request reference_audio_path / reference_text,
2. the avatar profile in data/voice_profiles.json (VoiceProfileStore, keyed by avatar_id),
3. the configured fallback reference (TTS_OMNIVOICE_FALLBACK_REFERENCE_AUDIO_PATH + TTS_OMNIVOICE_FALLBACK_REFERENCE_TEXT).

Only if even the fallback reference is invalid does it raise the non-recoverable MISSING_REQUIRED_FIELD. Prepared clone prompts are cached in-process, keyed by sha1(reference_audio_path + reference_text).

note

The per-avatar values originate in the gateway avatars table (voice_clone_path, voice_clone_text, elevenlabs_voice_id). The gateway forwards them into synthesis.start: the voiceover relay injects voice_clone_path / voice_clone_text, while the conversation path forwards elevenlabs_voice_id and the clone fields through the session config into the engine's TTSClient.

Configuration

Shared audio contract

Setting	Env var	Default	Purpose
sample_rate	TTS_SAMPLE_RATE	24000	Output PCM rate; a different requested value is rejected.
audio_format	TTS_AUDIO_FORMAT	pcm16le	Output format; a different requested value is rejected.
channels	TTS_CHANNELS	1	Mono output.
ws_path	TTS_WS_PATH	/ws/tts	Synthesis WebSocket route.

warning

max_text_chars (TTS_MAX_TEXT_CHARS) differs between backends: tts_service (ElevenLabs) defaults to 5000, while tts_omnivoice inherits the root default of 2000. A single TTS_MAX_TEXT_CHARS value cannot match both.

ElevenLabs (tts_service)

Setting	Env var	Default	Purpose
api_key	ELEVENLABS_API_KEY / ELEVENLABS_XI_API_KEY / XI_API_KEY	""	Auth; blank makes /health report degraded.
base_url	ELEVENLABS_BASE_URL	https://api.elevenlabs.io	API base URL.
voice_id	ELEVENLABS_VOICE_ID	JBFqnCBsd6RMkjVDRZzb	Default voice when the request has no elevenlabs_voice_id.
model_id	ELEVENLABS_MODEL_ID	eleven_multilingual_v2	Synthesis model.
output_format	ELEVENLABS_OUTPUT_FORMAT	pcm_24000	Stream output format.
timeout_sec	ELEVENLABS_TIMEOUT_SEC	30.0	HTTP client timeout.
voice settings	ELEVENLABS_STABILITY / ELEVENLABS_SIMILARITY_BOOST / ELEVENLABS_STYLE / ELEVENLABS_USE_SPEAKER_BOOST	0.5 / 0.8 / 0.0 / true	VoiceSettings sent on every stream call.

Supported language hints: en, pl, uk, tr, ru. Azerbaijani (az) maps to None (unsupported).

OmniVoice (tts_omnivoice)

Setting	Env var	Default	Purpose
model id	TTS_OMNIVOICE_MODEL_ID	k2-fsa/OmniVoice	HF model loaded via OmniVoice.from_pretrained.
device	TTS_OMNIVOICE_DEVICE	cuda:0	device_map.
dtype	TTS_OMNIVOICE_DTYPE	float16	Torch dtype (float16 / bfloat16 / float32).
num_step	TTS_OMNIVOICE_NUM_STEP	12	Default diffusion steps (per-request num_step clamped 1–50).
pcm_stream_ms	TTS_OMNIVOICE_PCM_STREAM_MS	20	PCM chunk size (ms) for streaming (min 10).
language_default	TTS_OMNIVOICE_LANGUAGE_DEFAULT	English	Fallback language for empty/unknown hints.
max_concurrent	TTS_MAX_CONCURRENT_SYNTHESIS	1	Global semaphore bound on concurrent GPU syntheses.
fallback ref audio	TTS_OMNIVOICE_FALLBACK_REFERENCE_AUDIO_PATH	<project_root>/reference_text.wav	Reference audio when neither request nor profile supplies a valid clone reference.
fallback ref text	TTS_OMNIVOICE_FALLBACK_REFERENCE_TEXT	"Captain Jim had often talked to Anne of Lost Margaret …"	Transcript paired with the fallback reference audio.

Voice profiles are loaded from data/voice_profiles.json (PATHS.voice_profiles_file). OmniVoice language mapping: en → English, az → Azerbaijani, tr → Turkish, ru → Russian; unknown hints default to language_default.

Engine wiring (fallback owner)

Setting	Env var	Default	Purpose
tts_ws_url	TTS_WS_URL	ws://localhost:8014/ws/tts	Primary (ElevenLabs) backend for FallbackTTSClient.
tts_timeout_sec	TTS_TIMEOUT_SEC	60.0	WS open_timeout for the primary.
tts_omnivoice_ws_url	TTS_OMNIVOICE_WS_URL	ws://localhost:8015/ws/tts	Fallback (OmniVoice) backend.
tts_omnivoice_timeout_sec	TTS_OMNIVOICE_TIMEOUT_SEC	120.0	WS open_timeout for the fallback.
tts_ws_url (gateway)	TTS_WS_URL	ws://localhost:8014/ws/tts	Gateway /api_lis/voiceover relay target — ElevenLabs only.