Conversation Pipeline

The conversation pipeline is the ordered sequence of stages the conversation_engine runs for every user turn. It lives in Pipeline.run_turn, an async generator that yields DomainEvent objects one at a time; the ConnectionHandler translates each event into an outbound WebSocket envelope. This page documents the exact stage order, which stages are fatal versus best-effort, and how image turns, grounding, prompt assembly, LoRA pinning, and fatal errors are handled.

For the wire-level event names each stage produces, see WebSocket Protocol. For the service that hosts the pipeline, see Conversation Engine.

Overview of a turn

A turn begins when the handler receives a message.text event and calls Pipeline.run_turn(session=..., message=..., image=...). run_turn is an async generator: it yields domain events as work progresses rather than returning a single result. The handler consumes the generator and, for each event, sends the corresponding wire envelope to the client.

Exactly one turn per session runs at a time. If a message.text arrives while a turn is active, the handler rejects it with REQUEST_ALREADY_IN_PROGRESS (recoverable). Each turn runs as an asyncio task named turn_{session_id}.

note

The pipeline never assembles a final answer itself beyond relaying LLM tokens. It orchestrates guard, persistence, memory, RAG, image analysis, prompt assembly, the LLM stream, and TTS — each behind a hexagonal port — and emits events describing what happened.

The stages

run_turn executes the following stages in order for a single message.text turn. The Failure mode column shows what happens when the stage fails.

#	Stage	What it does	Failure mode
0	Normalize + validate	resolve_question(text, image) picks the user text, or the default image question for image-only turns. If an image is present, validate_transport_image checks MIME, base64, and decoded size.	INVALID_IMAGE_PAYLOAD (recoverable) → turn ends
1	Guard check	guard.check(resolved_question) runs a safety classification.	Fail-open: any exception yields GUARD_ERROR (recoverable) but treats the request as safe
2	Safety gate	If the guard flags the request unsafe, emit response.started + response.completed with a refusal message (finish_reason='safety_filter') and return without persisting.	n/a (terminal, clean)
3	Persist user message	conversation_repo.append_turn(role='user', ...) writes to lis.conversation_history.	DB_ERROR (non-recoverable) → turn ends, session closed
4	Image analysis + attachment	Image turns only: save file + insert attachment row (best-effort), then run the VLM and emit image.analyzed; back-fill the attachment description.	Analyze exception or empty result → IMAGE_ANALYSIS_ERROR (recoverable) → turn ends. Storage/insert failure is swallowed
5	Parallel memory + RAG	asyncio.gather(return_exceptions=True) fetches memory (only when memory injection is on and a user_id is present) and RAG (always) concurrently.	Memory exception → MEMORY_ERROR (recoverable) + empty block. RAG exception → RAG_ERROR (recoverable) + low-confidence context
6	Prompt assembly	assemble_prompt(...) builds the flat system + user prompts and computes the grounding mode.	n/a
7	LLM stream	Emit response.started, then stream tokens as response.delta (last delta carries is_final=true), accumulating the full text.	LLM_ERROR (non-recoverable) → turn ends, session closed
8	Response completed	Emit response.completed with the final text, finish_reason='stop', low_confidence, citations, and (for image turns) image-analysis metadata including grounding_mode.	n/a
9	Persist assistant message	Best-effort append_turn(role='assistant', ...) (only when the text is non-empty).	Swallowed on failure
10	TTS synthesis	Best-effort, only when voice_output is on and text is non-empty: emit audio.synthesis_started, stream audio.chunk, always emit audio.completed.	All synthesis exceptions swallowed
11	Memory extraction	Fire-and-forget: schedules an asyncio.create_task when memory extraction is on, a user_id is present, and text is non-empty.	Detached; errors swallowed

Recoverable vs non-recoverable

Only two in-turn stages are non-recoverable: the user-message DB write (DB_ERROR) and the LLM stream (LLM_ERROR). An unhandled exception in the turn task surfaces as INTERNAL_ERROR (also non-recoverable). Every other in-turn error is recoverable — the client can retry without reconnecting.

Recoverable	Non-recoverable
INVALID_IMAGE_PAYLOAD, GUARD_ERROR, IMAGE_ANALYSIS_ERROR, MEMORY_ERROR, RAG_ERROR, LORA_NOT_READY, LORA_ERROR	DB_ERROR, LLM_ERROR, INTERNAL_ERROR

note

The guard is fail-open in two layers: the default guard client already swallows every exception and returns "safe", and run_turn additionally wraps the call. With the default adapter, GUARD_ERROR is effectively never emitted, and unsafe requests are only those the guard model explicitly flags. Blocked messages are not persisted — the safety gate returns before the DB write.

Image turns

Image turns come in two shapes: text + image (the user text is the question) and image-only (no text). For image-only turns, resolve_question substitutes a fixed default question:

Describe the relevant visible content in this image concisely.

Transport validation (validate_transport_image) accepts only image/jpeg, image/png, and image/webp (with image/jpg normalized to image/jpeg), requires valid base64, and enforces the decoded-size cap (IMAGE_ANALYSIS_MAX_IMAGE_BYTES, default 4 MiB).

Key rules for image turns:

• The VLM receives the image only — no question is forwarded to image analysis. The visual description it returns is question-agnostic.
• Raw image bytes never reach RAG or the final LLM. Only the extracted visual_text (truncated to 4000 chars) is placed into the prompt.
• The stored user message content is the resolved question. For image-only turns, that is the default question text above.
• Description back-fill. The attachment row is inserted with description = NULL, then updated with the VLM visual_text after analysis. If analysis fails or is dropped, the description stays NULL.
• Past image descriptions re-enter later turns through history, rendered as [Image: {description}] lines.

warning

Text found inside an image (OCR / in-image instructions) is explicitly marked as untrusted data in the prompt, never as instructions to follow. This is a prompt-injection mitigation baked into the image-turn policy and response rules.

Grounding modes

assemble_prompt chooses one of three grounding modes based on whether the turn has visual evidence and whether RAG produced usable, confident chunks:

Grounding mode	When	Effect on RAG chunks
rag_only	No visual evidence	RAG chunks included as Background Notes
visual_only	Visual evidence present and (RAG low-confidence or no chunks)	RAG chunks dropped, even if some were retrieved
visual_plus_rag	Visual evidence present and RAG returned confident chunks	Both visual evidence and RAG chunks included

The current image's visual_text is emitted under a VISUAL EVIDENCE FROM THE USER'S ATTACHED IMAGE section. The low-confidence fallback policy block is appended only when grounding_mode == 'rag_only' and RAG reported low confidence.

note

RAG in LIS 3 is retrieve-only: the RAG service returns ranked chunks and a low_confidence flag but never generates an answer. When RAG confidence collapses, the prompt narrows scope rather than fabricating facts. See the RAG service for retrieval internals.

Prompt assembly

assemble_prompt returns an AssembledPrompt with three fields: system_prompt, user_prompt, and grounding_mode. It produces two flat strings the LLM server expects:

• system_prompt — the avatar persona, optionally augmented with a [USER MEMORY] block (when memory was applied) and the image-turn policy (on image turns).
• user_prompt — assembled from the sections that carry content, in order: scope-control block, response rules (image vs text variant), grounding-mode line (image turns), conversation history, background notes (RAG chunks), visual evidence (image turns), the current user message, and — only in rag_only low-confidence turns — the fallback policy.

History passed to assembly is the tail of the in-memory session history excluding the current message: session.history[:-1][-context_window:].

LoRA pinning outside the turn

LoRA adapter pinning is not part of run_turn. It runs around the session lifetime:

• on_session_start loads recent history once into session.history, then normalizes the requested LoRA name and calls notify_starting. If the backend reports ready, the adapter is pinned (effective_lora_name set); if not ready, it yields LORA_NOT_READY (recoverable) and continues without LoRA.
• on_session_end always runs on close, cancel, and cleanup, calling notify_ending (best-effort) when an adapter was pinned.

The pinned effective_lora_name (or None) is passed into every llm.stream call for the session.

note

on_session_start runs only for brand-new sessions. A resumed session (same session_id still live in the in-memory store) skips history reload and LoRA re-pinning. Because the session store is process-local, resume only works within the same process while the session is still in memory.

Fatal error handling

When a turn produces a TurnError with recoverable == False, or an unhandled exception (sent as INTERNAL_ERROR), the handler marks the turn fatal. In the turn task's finally block it closes the generator, finishes the turn, and calls _close_session('fatal_error'), which:

1. Sends session.closed with reason: 'fatal_error'.
2. Removes the session from the in-memory LiveService store.
3. Closes the WebSocket (websocket.close()).

warning

On a fatal error the WebSocket connection is closed and the in-memory session is removed. The client must reconnect and send a fresh session.start to continue. (Some older prose claims the socket stays open — the code closes it.)

Recoverable errors do not close the session: the client receives the error envelope and can send another message.text on the same connection.