Skip to main content

AMADEQ LIS 3

AMADEQ LIS 3 is a Live Interaction System — a multi-service platform for real-time, multimodal conversation with AI avatars. It combines streaming text chat, speech-to-text, text-to-speech, retrieval-augmented grounding (RAG), long-term memory, image understanding, a safety guard, and per-avatar LoRA adapters behind a single authenticated gateway.

The platform is built in a clean, hexagonal architecture: each service isolates its transport layer from transport-agnostic application logic, which depends only on port interfaces backed by injected adapters (HTTP/WebSocket clients, database repositories, model backends). This keeps every service independently testable and independently deployable.

System at a glance​

A browser or client speaks only to the gateway. The gateway authenticates the caller and relays WebSocket traffic to the conversation engine and the STT/TTS services. The conversation engine orchestrates each turn, reaching component services and shared infrastructure through its ports.

                          client (browser / app)
                                   │
                                   │  HTTP + WebSocket  (prefix /api_lis)
                                   ▼
                        gateway_service        :8040
                        auth · REST · WS relay
                                   │
          ┌────────────────────────┼───────────────────────────┐
          │ WS                      │ WS                          │ WS
          ▼                         ▼                             ▼
  conversation_engine :8050   stt_service :8010         tts_elevenlabs :8014  (primary)
  WS /message_pipeline        WS /ws/stt                tts_omnivoice  :8015  (fallback)
          │
          │  per-turn orchestration (HTTP unless noted)
          ├───────────────► memory_service          :8052  GET /get_injection_context · POST /extract
          ├───────────────► rag_service             :8051  POST /retrieve
          ├───────────────► image_analysis_service  :8012  POST /analyze  ──► vLLM VLM backend :8013
          ├───────────────► guard                   :4915  OpenAI-compatible safety check
          └───────────────► LLM                     remote OpenAI-compatible streaming

  shared infrastructure
          ├── PostgreSQL (Neon)   lis.sessions · lis.conversation_history · lis.conversation_attachments
          │                        · lis.memory_facts · ml.avatars (read-only)
          ├── Qdrant       :6333  vector store for RAG collections
          ├── embeddings   :8080  TEI · intfloat/multilingual-e5-large
          └── reranker     :8081  TEI · BAAI/bge-reranker-v2-m3
note

All gateway HTTP and WebSocket routes mount under the /api_lis prefix. The gateway's GET /health liveness probe is the one exception — it is served at the root.

Services​

The platform runs eight independently launched services. Ports are the defaults bound by run/.

NamePortPurpose
gateway_service8040User-facing HTTP + WebSocket gateway: authentication, REST CRUD for sessions and memory facts, and an authenticated WS relay to the engine and STT/TTS services.
conversation_engine8050Live text-chat WebSocket (/message_pipeline); orchestrates each turn — guard, persistence, image analysis, memory + RAG, LLM streaming, TTS.
stt_service8010Streaming speech-to-text over WS /ws/stt, backed by faster-whisper (English-forced in this deployment).
tts_elevenlabs8014Primary text-to-speech backend (ElevenLabs cloud) over WS /ws/tts.
tts_omnivoice8015Fallback text-to-speech backend (local OmniVoice GPU model with reference-audio voice cloning) over WS /ws/tts.
image_analysis_service8012Stateless HTTP adapter (POST /analyze) for question-agnostic visual-context extraction; fronts a separate vLLM VLM backend on port 8013.
memory_service8052LLM-driven long-term user memory: injection context (GET /get_injection_context) and fact extraction (POST /extract).
rag_service8051Retrieve-only RAG pipeline (POST /retrieve): query rewrite, embed, Qdrant search, rerank, confidence filtering.

Shared infrastructure​

  • PostgreSQL (Neon) — one database holding lis.sessions, lis.conversation_history, lis.conversation_attachments, and lis.memory_facts, plus the read-only ml.avatars table.
  • Qdrant (port 6333) — vector store for the per-avatar RAG collections searched by rag_service.
  • embeddings (TEI) (port 8080) — Text Embeddings Inference server running intfloat/multilingual-e5-large; used by RAG and memory.
  • reranker (TEI) (port 8081) — Text Embeddings Inference server running BAAI/bge-reranker-v2-m3; reranks RAG candidates.
  • guard (port 4915) — OpenAI-compatible safety model (Qwen/Qwen3Guard-Gen-0.6B on vLLM) that the conversation engine calls fail-open before each turn.

Key capabilities​

  • Streaming text chat — token-by-token response streaming over a single WebSocket, with one active turn per session and cooperative cancellation.
  • Speech-to-text — real-time partial and final transcripts with energy-based plus model VAD endpointing.
  • Text-to-speech — streamed PCM16LE audio at 24 kHz, with an ElevenLabs primary backend and a transparent OmniVoice fallback owned by the conversation engine.
  • RAG grounding — retrieve-only pipeline with per-avatar and per-user collections, reranking, and a low-confidence signal that drives grounding-mode selection.
  • Long-term memory — LLM extraction of durable user facts with embedding-based deduplication, plus budgeted memory injection into the prompt.
  • Image understanding — validated image uploads analyzed by a VLM into bounded visual text, persisted as conversation attachments.
  • Safety guard — a fail-open safety check gates every turn; blocked messages return a refusal and are not persisted.
  • Per-avatar LoRA adapters — the engine pins and releases a LoRA adapter around each session's lifetime.

The avatars that LIS 3 serves are produced by a separate system: Onboarding V4, the creator-facing onboarding and LoRA training platform. It turns a creator's specialty description and writing samples into a per-avatar LoRA adapter, a runtime system prompt, and a voice selection, then writes them to the shared ml.avatars table that LIS 3 reads. The two systems share only that table — Onboarding V4 owns it, LIS 3 only reads from it.

Next steps​