WebSocket Protocol

The gateway exposes three authenticated WebSocket endpoints, all mounted under the /api_lis prefix. Each is a relay: the gateway authenticates the caller, resolves session and avatar data from the database where needed, and forwards traffic to an upstream component service (conversation engine, STT, or TTS). This page documents the wire protocol for all three.

Authentication is passed as query parameters on the connection URL: ?user_id=<uuid>&api_key=<key>. The API-key check is fail-open — when the gateway has no key configured (GATEWAY_API_TOKEN empty), no api_key is required. See Conversation Pipeline for what happens after the message relay reaches the engine.

Common envelope

Every WebSocket message — inbound and outbound — shares the same envelope shape:

{
  "event": "<event_name>",
  "session_id": "<string>",
  "timestamp": "2024-01-01T12:00:00.000Z",
  "request_id": "<string | null>",
  "payload": {}
}

Field	Type	Notes
event	string	Event name (e.g. message.text, response.delta).
session_id	string	Session identifier. On inbound messages it may be omitted before a session exists.
timestamp	string	ISO-8601 UTC (Z suffix). Set automatically on outbound messages.
request_id	string | null	Optional correlation id. When set on an inbound message, it is echoed on every response event for that request.
payload	object	Event-specific body.

note

Outbound envelopes may also carry a metadata object (used by the STT service for per-message model/timing info). request_id is only echoed when the client supplied one.

WS /api_lis/sessions/{session_id}/message

Relays a conversation turn to the conversation engine. The gateway resolves the session's avatar before connecting and injects the avatar's system prompt, voice-clone reference, LoRA adapter name, and collection into the upstream session.start event automatically. Raw image bytes never reach the LLM.

Connection URL

ws://<host>/api_lis/sessions/<session_uuid>/message?user_id=<uuid>&api_key=<key>

First frame: UserPreferences

Before any event exchange, the client must send a UserPreferences JSON object as the very first message (this is not an event envelope):

{
  "language_hint": "en",
  "memory_injection": true,
  "memory_extraction": true,
  "voice_output": false
}

Field	Type	Default	Notes
language_hint	string | null	null	Preferred response language.
memory_injection	bool	true	Inject stored memory facts into the prompt.
memory_extraction	bool	true	Extract new facts from the turn (fire-and-forget).
voice_output	bool	false	Synthesize the assistant reply to audio.

If the first frame is not valid UserPreferences JSON, the socket is closed with code 4003. After this frame the connection becomes a transparent bidirectional relay to the conversation engine.

Inbound events (client → engine)

Event	Payload	Notes
message.text	See below	Sends a user turn. Empty text is rejected. Optional image attaches an image.
session.ping	{}	Keepalive.
session.cancel	{"reason": "user_interrupt"}	Cancels the running turn.
session.close	{}	Closes the session gracefully.

message.text payload:

{
  "text": "What is in this image?",
  "image": {
    "mime_type": "image/jpeg",
    "data_base64": "<base64-encoded bytes>",
    "filename": "photo.jpg"
  }
}

image is optional; filename inside image is optional. Supported MIME types: image/jpeg, image/png, image/webp (image/jpg is accepted and normalized to image/jpeg). When present, the image is saved to attachment storage, analyzed by the VLM service, and the resulting description is added to the LLM prompt context.

Outbound events (engine → client)

Event	Payload	Notes
error	{"code": "...", "message": "...", "recoverable": true}	See error codes below. May appear during session start (e.g. LORA_NOT_READY), before session.started.
session.started	{"session_id": "...", "avatar_id": "...", "resumed": false}	Emitted after session setup completes.
message.safety_check	{"safe": true, "reason": null}	Result of the safety guard, emitted before the response.
memory.injected	{"facts_count": 3}	Emitted only when memory facts were applied.
rag.retrieved	{"chunks_count": 4, "low_confidence": false}	Emitted when RAG context was retrieved.
image.analyzed	{"latency_ms": 320.5, "model_id": "...", "analysis_type": "..."}	Emitted after the VLM analyzes an attached image, before the LLM turn.
response.started	{"status": "started"}	Emitted before every response, including safety-filtered replies.
response.delta	{"delta": "Hi ", "is_final": false}	Streamed text token. Only the last token carries is_final: true.
response.completed	{"text": "...", "finish_reason": "stop", "metadata": {"low_confidence": false, "citations": []}}	Full assembled reply. finish_reason is "safety_filter" when the guard blocked the request; metadata may also include image_analysis.
audio.synthesis_started	{}	Emitted when voice_output: true.
audio.chunk	{"chunk_index": 0, "audio": "<base64>", "audio_format": "pcm16le", "sample_rate": 24000, "channels": 1, "is_final": false, "duration_ms": 0}	PCM audio frame.
audio.completed	{}	All audio frames sent.
session.pong	{"status": "alive"}	Response to ping.
session.cancelled	{"reason": "user_interrupt"}	Sent immediately when cancel is received; the turn stops concurrently.
session.closed	{"reason": "client_close" | "fatal_error" | "idle_timeout"}	Session closed; the connection closes after this.

Error codes

Code	Recoverable	Meaning
SESSION_NOT_STARTED	false	An event other than session.start arrived first.
INVALID_EVENT	true	Unknown event type.
INVALID_PAYLOAD	true	Malformed or missing payload field.
REQUEST_ALREADY_IN_PROGRESS	true	A turn is already running; cancel it first.
INVALID_IMAGE_PAYLOAD	true	Image MIME, base64, or size validation failed.
GUARD_ERROR	true	Safety guard check failed; treated as safe (fail-open).
DB_ERROR	false	Persisting the user message failed.
IMAGE_ANALYSIS_ERROR	true	VLM analysis failed or returned empty text.
LORA_NOT_READY	true	Requested LoRA adapter not ready; continues on the base model. Emitted during session start.
LORA_ERROR	true	LoRA start notification failed; continues on the base model. Emitted during session start.
MEMORY_ERROR	true	Memory injection failed; continues without injected facts.
RAG_ERROR	true	RAG retrieval failed; continues without retrieved context.
LLM_ERROR	false	LLM streaming failed.
INTERNAL_ERROR	false	Unhandled server error.

warning

A non-recoverable error closes the session: the engine sends session.closed with {"reason": "fatal_error"} and then closes the socket. The client must reconnect and send a new UserPreferences frame to continue.

WS /api_lis/transcribe

Relays raw audio to the STT service for real-time transcription. The gateway is a pass-through; it only appends user_id and client_type=transcribe (plus language_hint when provided) to the upstream URL.

Connection URL

ws://<host>/api_lis/transcribe?user_id=<uuid>&api_key=<key>&language_hint=<lang>

language_hint is optional.

note

Recognition language is hard-forced to English in this deployment. Any client-supplied language_hint is ignored by the STT service and every transcript reports language: "en".

Inbound events (client → STT)

Event	Payload	Notes
session.start	{"session_id": "...", "audio_format": "pcm16le", "sample_rate": 16000}	Must be first. audio_format must be pcm16le and sample_rate must be 16000; mono only.
audio.chunk	{"chunk_index": 0, "audio": "<base64 PCM16LE>", "sample_rate": 16000, "channels": 1}	Raw audio frame. chunk_index must be strictly increasing.
audio.end	{}	Signals end of audio; forces a final transcript.
session.cancel	{"reason": "..."}	Cancels the current utterance and closes the socket.
session.ping	{}	Keepalive.

Outbound events (STT → client)

Event	Payload	Notes
session.started	{"status": "ready", "model": "...", "language_mode": "forced_en", ...}	Session ready. Also carries a read-only config snapshot (timeouts, decode/partial intervals, VAD threshold, preroll, max utterance).
transcript.partial	{"text": "...", "stability": 0.9, "language": "en", "start_ms": 0, "end_ms": 1200, "chunk_index": 7, "is_final": false}	In-progress result. No utterance_id.
transcript.final	{"text": "...", "language": "en", "confidence": 0.97, "start_ms": 0, "end_ms": 2400, "utterance_id": "...", "is_final": true, "finalize_reason": "silence_timeout"}	Committed utterance.
transcript.empty_final	{"utterance_id": "...", "finalize_reason": "...", "detail": "no_audio_detected" | "empty_transcript", "is_final": true}	Emitted when an utterance has no audio or produced empty text.
session.pong	{"status": "alive"}	Response to ping.
session.cancelled	{"reason": "..."}	Session cancelled (also on idle timeout, with reason: "timeout").
error	{"code": "...", "message": "...", "recoverable": true}	Service error.

WS /api_lis/voiceover

Relays a TTS synthesis request to the TTS service (ElevenLabs backend). The gateway intercepts the first synthesis.start frame and injects the session avatar's voice-clone reference into it. This path has no OmniVoice fallback.

Connection URL

ws://<host>/api_lis/voiceover?user_id=<uuid>&api_key=<key>&session_id=<session_uuid>

session_id is required — it is used to look up the avatar's voice-clone data. A missing or invalid session_id closes the socket with code 4003; a session not owned by the caller closes with 4004.

Inbound: synthesis.start

The client's first message must be a synthesis.start event:

{
  "avatar_id": "<string>",
  "text": "Hello world",
  "stream": true,
  "language_hint": "en",
  "elevenlabs_voice_id": "<string | null>",
  "reference_audio_path": "<string | null>",
  "reference_text": "<string | null>"
}

stream must be true; audio_format (if given) must be pcm16le and sample_rate 24000. The gateway sets reference_audio_path and reference_text from the avatar record only when those keys are absent — client-provided values are preserved. session.ping and session.cancel are also accepted.

Outbound events (TTS → client)

Event	Payload	Notes
synthesis.started	{"status": "started", "avatar_id": "...", "stream": true, "request_state": "streaming_elevenlabs", "backend": "elevenlabs"}	Synthesis has begun.
audio.chunk	{"chunk_index": 0, "audio": "<base64>", "audio_format": "pcm16le", "sample_rate": 24000, "channels": 1, "is_final": false, "duration_ms": 80, "backend": "elevenlabs"}	PCM audio frame.
synthesis.completed	{"total_chunks": 12, "total_audio_duration_ms": 960}	All audio delivered.
error	{"code": "...", "message": "...", "recoverable": true}	Service error.

note

A voice.resolved event is emitted before the audio stream only when the synthesis.start payload sets debug: true.

WebSocket close codes

Code	Meaning
4001	Auth failure (bad/missing api_key, or missing user_id).
4003	Malformed id, or an invalid first frame.
4004	Session or avatar not found / not owned by the caller.
1014	Upstream component service connection failed.