Testing

The suite is a flat pytest package of 16 test files (~1,900 lines) covering the workflow service, domain matching, configuration integrity, the training chain, and the voice roster. No test contacts PostgreSQL, Redis, OpenRouter, a GPU provider, or ElevenLabs: the database is replaced by hand-written fakes, and every network boundary is monkeypatched.

Running tests

pytest tests -q

tests/conftest.py — the only conftest — inserts the repository root into sys.path, loads .env with override=False, and defaults DB_SCHEMA to ml. No PYTHONPATH is needed. requirements.txt pins pytest==8.3.5.

warning

Several tests read the real state/config/ tree — the catalog, the base datasets, the evaluation questions, the checksum manifest, and the candidate packs. They fail on a checkout with a missing or drifted config tree. Run python -m domain.integrity --config-root state/config first if the domain tests fail unexpectedly.

.env is loaded but not required. Only the schema name is forced; no test opens a connection.

Test layout

Application surface

File	Verifies
test_app.py	FastAPI metadata (Onboarding V4 Workflow API, 4.0.0); a non-UUID avatar_id on POST /v1/runs is a 422; the route table matches the supported production contract; character_story distinguishes omitted / set / cleared and enforces its 6,000-character budget; avatar deletion returns the soft-delete result, 404s for a non-owned avatar, and rejects an invalid id.
test_onboarding_routes.py	GET /health returns the workflow status payload; the static onboarding page is served.
test_onboarding_service.py	WorkflowCore against a FakeDB and a FakeDomainCore: an uncertain match forces domain_confirmation_required; too few seeds force seed_examples_pending before queueing; confirm_domain rejects a wrong status; get_generated_samples raises before samples exist; confirmation syncs the avatar's catalog fields; a failing initialization marks the run failed; a valid ElevenLabs voice id is persisted and an invalid one rejected; character_story is persisted and cleared; a soft-deleted avatar disappears from reads and updates; deletion enforces creator ownership.

Domain configuration and matching

File	Verifies
test_domain_catalog.py	The real catalog loads and is non-empty.
test_domain_assignments.py	The real avatar mapping loads as a dict.
test_domain_matcher.py	An override bypasses embedding; no catalog match forces confirmation; an insufficient runner-up margin blocks auto-assignment even at a high score; a clear high-score winner auto-assigns; a TEI failure propagates rather than degrading; catalog fields map to avatar fields; parallel confirm_assignment writes do not corrupt the mapping file.
test_domain_integrity.py	The repository's own domain configuration is internally consistent; the checksum manifest covers every dataset and evaluation file; --fix normalization replaces absolute dataset paths with portable filenames and refreshes sample counts.
test_domain_candidate.py	Every shipped candidate pack passes the validator; the validator detects evaluation leakage into training seeds; the candidate generation contract builds exactly 800 examples; the generation prompt never asks the model for free-form RAG context.

Generation and review

File	Verifies
test_generate_samples.py	The generation system prompt asks for both STANDALONE and RAG blocks.
test_review_samples.py	review_approved → postprocess_running is an allowed transition.
test_finalize_dataset.py	The smart.finalize Celery task is registered and callable.

Training

File	Verifies
test_training_service.py	A UUID avatar_id survives dataset-payload serialization; build_domain_questions selects the expected count; dataset and question lookups are case-insensitive; the judge's JSON payload parser handles the single response shape; answer pairs keep question order despite concurrent inference.
test_training_tasks.py	The inline path includes the terminate stage; start_training refuses a terminal run before creating a job; the dataset payload prefers final_dataset.examples and fails without it; _patch_run_status refuses to progress a terminal run; run_stage_complete evaluates the LoRA and updates the avatar; artifact sync downloads optional files and writes locally while ignoring remote-artifact env overrides; the avatar profile stores the adapter directory as the DB LoRA path; best-effort termination updates the manifest; the request stage populates instance env, prefers RunPod, and falls back to Vast.ai when RunPod is unavailable; endpoint selection falls back to the cached endpoint when the refreshed port is bad, and the retry helper reuses it without consuming a Celery retry; the Vast client searches an offer before creating an instance.

Voice

File	Verifies
test_voice_service.py	The default roster has ten voices per gender; preview generation requires a WAV output format; GET /v1/voice-previews returns the full default roster.

Contract parity

test_production_flow_parity.py is a regression lock against the deployed system. It asserts, against hard-coded expected values:

• the complete RUN_STATUSES tuple, in order, including the legacy training_ready;
• that the happy path is walkable through ALLOWED_TRANSITIONS;
• the exact order of the eight training.* tasks in the chain;
• that the generation system and user prompts, the avatar-prompt builder, the fallback prompt, and all three LoRA-evaluation prompt builders still produce the production wording;
• that the embedding input helper still applies the E5 passage: prefix.

Changing a prompt string, a status name, or the stage order will fail this file. That is intentional — the assertions are the record of what production expects, so update them deliberately and in the same commit as the behaviour change.

Fakes

There is no shared fixture module; each test file defines what it needs inline.

Fake	Defined in	Stands in for
FakeDB	test_onboarding_service.py	In-memory runs, jobs, events, and avatars implementing the DBStore surface WorkflowCore touches, including status transitions and soft delete.
FakeDomainCore	test_onboarding_service.py	Domain matching with a scripted decision; BrokenConfirmDomainCore raises on confirmation to exercise the failure path.
FakeDB	test_training_tasks.py	Runs, jobs, and the artifact manifest for the training stages, with merge semantics.
StubDB	test_voice_service.py	Minimal store so the app factory can be built for the previews endpoint.

Provider clients, SSH helpers, embedding calls, and OpenRouter clients are replaced per test with monkeypatch.

What is not covered

• No end-to-end test crosses the Celery boundary — tasks are exercised by calling the underlying core methods, and queue routing itself is untested.
• No test exercises a live Kimi generation, real SSH transport, or train_unsloth.py; the remote training script has no test coverage at all.
• The review-window timers (smart.review_reminder_24 / _48, smart.finalize_review) and the review_epoch guard have no dedicated tests.
• The polling worker_runtime and its three repair passes are untested.