Testing
The test suite is a pure-Python pytest tree that mirrors the three deployable layers of the system: the gateway service, the conversation engine, and the four stateless component services. Every suite drives real application code through in-process factories, with database, model, and network dependencies replaced by hand-written fakes.
Running tests
Run the whole suite from the repository root:
PYTHONPATH=. pytest tests -q
There is no pytest.ini, pyproject.toml, setup.cfg, or tox.ini in the tree, so PYTHONPATH=. is required to make the application packages importable. requirements.txt pins pytest>=8,<9.
Every application factory (build_gateway_app, build_live_app, build_image_analysis_app, build_memory_app, build_rag_app, build_tts_app) is invoked with enable_db=False, which skips the database connection-pool lifespan. As a result the suite contacts no Postgres, Qdrant, ElevenLabs, or GPU backend.
The gateway and memory factories are additionally built with api_key=''. An empty key disables X-API-Key enforcement, so HTTP tests authenticate with only the X-User-ID header. The API-key check fires only when the expected key is truthy.
tests/test_conversation_engine/conftest.py is the only conftest.py; it is scoped to that package. The other two packages have no conftest, so their fixtures and factory helpers are defined inline inside each test file. The __init__.py files in each package are empty.
Test layout
The tree has three packages, one per layer. Each table below maps a test file to what it verifies.
test_gateway_service/
Covers the FastAPI HTTP/WS gateway. Its routers mount under the /api_lis prefix.
| File | Verifies |
|---|---|
test_session.py | Session routes: GET/POST /api_lis/sessions, PATCH/DELETE /api_lis/sessions/{session_id}. Create with a known avatar → 201; unknown avatar → 404; rename/soft-delete → 204; not-found or wrong-owner → 404. |
test_message.py | GET /api_lis/sessions/{session_id}/messages returns {session_id, messages[], count} in chronological order; missing or other-owner session → 404. |
test_memory.py | GET /api_lis/memory lists the user's active facts and filters out other users'; DELETE /api_lis/memory/{fact_id} → 204, malformed UUID → 400. |
test_clients.py | Outbound WS relay clients: build_url query merging; ConversationEngineClient / STTClient / TTSClient frame shaping; upstream-connect failure closes the downstream client with code 1014. |
Gateway auth (get_current_user in gateway_service/api/dependencies.py) returns HTTP 400 for a missing X-User-ID header and 400 for a malformed UUID (including a bad path segment). Ownership is enforced by returning 404, not 403, when a resource belongs to another user.
test_conversation_engine/
Covers the WebSocket live-conversation orchestrator. All of these connect to the root-level /message_pipeline WS route (not under /api_lis).
| File | Verifies |
|---|---|
test_session.py | Session lifecycle: session.start → session.started (resumed=False even on reconnect); session.ping → session.pong; session.close → session.closed; disconnect removes the session; message before start → SESSION_NOT_STARTED; unknown event → INVALID_EVENT; whitespace-only text → INVALID_PAYLOAD; session.cancel with no running turn → session.cancelled. |
test_text_turns.py | Text-turn event order message.safety_check → response.started → response.delta(s) → response.completed. Safe turns finish with stop; guard-blocked turns finish with safety_filter and write nothing to the repo. A forced repo write failure yields error DB_ERROR with recoverable=False. |
test_delta_whitespace.py | Streaming delta invariants: space tokens preserved verbatim; ResponseCompleted.text rstrips only trailing newlines; concat(deltas).rstrip('\n') == completed.text; is_final true only on the final delta. |
test_history.py | History is fetched exactly once at session start (get_recent_turns_calls == 1 after two turns), not per turn; the current user message appears once in the assembled prompt. |
test_image_turns.py | Image-turn order and gating: image.analyzed before response.started; invalid transport → INVALID_IMAGE_PAYLOAD before guard/VLM/RAG run; safety block stops before VLM and RAG; image-only turns substitute DEFAULT_IMAGE_QUESTION; RAG is fail-open after a successful VLM step (RAG_ERROR, turn continues visual_only). |
test_attachments.py | Attachment persistence is best-effort and independent of VLM: the file is saved and the row inserted with description=None, back-filled only after successful analysis; empty visual_text still saves but ends on IMAGE_ANALYSIS_ERROR. |
test_lora.py | LoRA lifecycle: a ready adapter emits no pre-session error; a not-ready adapter emits LORA_NOT_READY (recoverable=True) before session.started (the session still starts); both explicit close and abrupt disconnect trigger notify_ending exactly once. |
test_voice.py | voice_output=True adds audio.synthesis_started and audio.completed after response.completed; voice_output=False emits none of the audio events. |
test_application_modules.py | Unit tests for normalize_lora_name, the safety message, and assemble_prompt, including grounding-mode selection (rag_only, visual_only, visual_plus_rag) and the low-confidence fallback-policy block. |
IMAGE_ANALYSIS_ERROR is recoverable (recoverable=True) and skips RAG/LLM, whereas DB_ERROR is fatal (recoverable=False). This distinction is asserted directly in the engine suites.
test_component_services/
Covers the four stateless model microservices. Each app is root-mounted (no /api_lis prefix).
| File | Verifies |
|---|---|
test_image_analysis.py | validate_image rejects unsupported MIME, non-base64, magic-byte/MIME mismatch, and truncated images; POST /analyze → {visual_text, model_id, latency_ms}, 400 bad MIME, 422 if a caller supplies the question field, 503 when the provider is not ready; GET /health → 200/503. |
test_memory.py | POST /extract saves parsed facts, dedups identical transcripts, 422 on blank transcript, 503 when the extraction LLM raises; GET /get_injection_context returns applied=false when empty and applied=true after extraction. |
test_rag.py | POST /retrieve → {context_chunks, citations, low_confidence}; low_confidence=true with zero chunks when only weak chunks match; 422 blank query; 404 missing collection; 503 embedder failure; optional additive user_collection_name falls back to base-only. |
test_tts.py | GET /health reports service='tts_service', backend='elevenlabs'; WS /ws/tts on synthesis.start(stream=true) → synthesis.started → audio.chunk (base64 PCM, sample_rate 24000) → synthesis.completed; stream=false → INVALID_PAYLOAD; session.ping → session.pong. |
Test doubles
The suite substitutes hand-written fakes for every external dependency. They fall into three groups.
In-memory repositories — tests/test_gateway_service/fake_repos.py provides FakeAvatarRepo, FakeSessionRepo, FakeMessageRepo, and FakeMemoryRepo. Each backs the real gateway_service.db.models rows with a dict, exposes a synchronous seed() for setup, and enforces the same ownership and soft-delete rules as the production repositories. Factory helpers make_avatar, make_session, and make_memory_fact build valid model instances for seeding.
Stub WS clients — tests/test_gateway_service/fake_clients.py provides FakeConversationEngineClient, FakeSTTClient, and FakeTTSClient. These no-op relay clients are injected as the gateway's upstream conversation_engine / stt / tts dependencies so HTTP tests never open real upstream sockets. test_clients.py additionally defines FakeClientWS, a minimal Starlette-WebSocket stand-in that records sent frames and a close code and replays a scripted receive() queue.
Conversation-engine fixtures — tests/test_conversation_engine/conftest.py wires the engine's ports to fakes and exposes the build_engine(**kwargs) factory, which builds build_live_app(enable_db=False) and returns a (TestClient, LiveService) pair; keyword arguments override individual fakes. The port fakes include:
| Fake | Role |
|---|---|
FakeGuard | Safety port; flags bomb / kill yourself / self-harm as unsafe. |
FakeLLM | Streaming LLM yielding a mock response token by token; records prompts. |
FakeRAG / FailingRAG | RAG port returning a configurable context, or raising to exercise the fail-open path. |
FakeImageAnalysis / FailingImageAnalysis / EmptyVLM | VLM port returning canned analysis, raising, or returning empty visual text. |
FakeMemory / FakeTTS | No-op memory port and an empty-audio TTS port. |
FakeConvRepo | Conversation repo with a fail_with hook and a call counter to assert history is fetched once. |
FakeAttachmentRepo / FakeAttachmentStorage | Record attachment insert() and storage save() calls. |
FakeLoRA | LoRA port with a configurable ready flag; records notify_starting / notify_ending. |
The conftest also provides the WS helpers ws_start, ws_start_with_lora, and collect_until (which drains events until a named stop event), plus the _FakeMessage and _FakeAttachment dataclass stand-ins for DB rows.
The test_clients.py relay tests are the only ones that open a real network socket: they start a localhost websockets server on an ephemeral port (0) and drive the real gateway relay clients against it, pointing at ws://localhost:1 to force an upstream-connect failure.
For guidance on adding new tests and fakes when you extend a service, see Extending the System.