Onboarding V4 Overview

Onboarding V4 (the Smart Training System) is the creator-facing side of the platform: it turns a short specialty description plus a batch of the creator's own writing samples into a trained, quality-scored per-avatar LoRA adapter and a runtime system prompt. Where AMADEQ LIS 3 serves avatars in live conversation, Onboarding V4 produces them.

The two systems meet in one place only: the ml.avatars table. Onboarding V4 owns it end to end — persona name, description, tags, system_prompt, lora_adapter_path, voice settings, and LoRA quality verdict. LIS 3 reads a single row by UUID and never writes to it.

System at a glance

Onboarding V4 is a single FastAPI application plus two Celery worker queues over one PostgreSQL schema and a local runtime-state tree. Everything heavy — dataset generation, GPU rental, remote training, adapter evaluation — happens off the request path in Celery tasks.

                       creator (browser / static client)
                                     │
                                     │  HTTP  /v1 · /api · /client
                                     ▼
                        app.py — Onboarding V4 Workflow API
                        FastAPI · WorkflowCore                   :8094
                                     │
                    ┌────────────────┴──────────────────┐
                    │ Redis (Celery broker + backend)   │
                    ▼                                   ▼
          queue "smart"                         queue "training"
   smart.generate · smart.finalize        training.request_instance
   smart.review_reminder_24 / _48         training.wait_instance_ready
   smart.finalize_review                  training.upload_payload
                    │                     training.start_remote
                    │                     training.monitor
                    │                     training.sync_artifacts
                    │                     training.terminate_instance
                    │                     training.complete
                    │                                   │
                    ▼                                   ▼
   OpenRouter (Kimi K2.5)                  RunPod  (primary GPU provider)
   dataset + prompt generation             Vast.ai (fallback provider)
   TEI embeddings  :8017                   SSH → train_unsloth.py on the pod
   domain matching · dedup · style         core LLM /api/generate (LoRA eval)
                    │                                   │
                    └───────────────┬───────────────────┘
                                    ▼
                   PostgreSQL schema "ml"       state/ (runtime tree)
                   app_runs · app_jobs          runs/ · models/ · config/
                   app_events · avatars         voice/ · secrets/ · logs/

Processes

ops/start_all.sh launches three processes and writes their PIDs to state/pids/.

Process	Command	Purpose
api	uvicorn app:app	The Workflow API. Initializes the ml schema on startup and validates the voice-preview manifest.
smart_worker	celery -A training.celery_worker:celery_app worker -Q smart --pool threads	Dataset generation, finalization, and the review-window timers.
training_worker	celery -A training.celery_worker:celery_app worker -Q training --pool threads	The eight-stage remote training chain.

note

training/tasks.py contains a fourth, optional process — a polling worker_runtime that claims jobs straight from app_jobs instead of Redis, and that repairs orphaned or stale jobs. It is not started by ops/start_all.sh, and its inline job processing is off unless WORKER_RUNTIME_INLINE_ENABLED=true. See Run Lifecycle.

The creator flow

One app_runs row tracks the whole journey. The API returns a next_action field on every run response so a client never has to infer the current step.

1. Create a run — POST /v1/runs with specialty_text, 20–50 seed_examples, an avatar_id, and an optional ElevenLabs voice. The specialty text is embedded and matched against the domain catalog.
2. Confirm the domain — only when the match is not confident enough (next_action: confirm_domain).
3. Submit seed examples — only when they were not supplied at creation (next_action: submit_seed_examples).
4. Smart generation — a Celery job asks Kimi K2.5 for 5 personalized examples per seed and rewrites the domain base prompt into an avatar-specific runtime system prompt.
5. Review — the creator approves, requests an edit, or rejects the generated samples. Reminders fire at 24 h and 48 h; at 72 h the run auto-approves.
6. Finalize — the approved samples are style-scored against the domain pack, oversampled, merged with the pack, and deduplicated into one final dataset.
7. Train — a GPU instance is rented, the dataset and training script are uploaded over SSH, train_unsloth.py runs detached, and the resulting adapter bundle is pulled back to state/models/{avatar_id}/.
8. Evaluate — the adapter answers 20 held-out domain questions head-to-head against the base model; an LLM judge scores each pair and the verdict lands on the avatar row.

Capabilities

• Embedding-based domain matching — 26 catalogued domains, matched by cosine similarity against multiple hand-written descriptions per domain, with both an absolute-score and a runner-up-margin gate before auto-assignment.
• Kimi-only dataset generation — per-seed parallel requests with partial-progress retries and exact per-seed quotas; mock and local endpoints are actively rejected.
• Style-aware dataset blending — the personal samples are compared to the domain pack by embedding centroid, and oversampled 1.5×–3× depending on how distinct the creator's style is.
• Two-provider GPU training — RunPod first, with automatic Vast.ai fallback when RunPod reports no capacity, and best-effort instance termination on every failure path.
• Resumable training chain — each of the eight stages is idempotent and rank-ordered, so a crashed chain can be re-dispatched from the last recorded pipeline status.
• Automated LoRA quality gate — 20 questions × (base + LoRA) answers, judged for accuracy, hallucination, and improvement-over-base, with a hard pass threshold.
• Auditable runs — every state change appends to ml.app_events, and every summary is also written as JSON under state/runs/{run_id}/.
• Soft-deleted avatars — deletion flips status to deleted; the row is retained and excluded from all reads.

Deliberate exclusions

Knowledge ingestion (source upload, chunking, Qdrant indexing) and speech-to-text are not part of Onboarding V4, even though ml.app_sources, ml.app_source_contents, and ml.app_source_chunks exist in the schema. Those tables are created but unused by this codebase.

Next steps

• Setup and Operations — install, environment, start/stop, schema and config validation.
• Run Lifecycle — statuses, transitions, jobs, and the review window.
• API Reference — every HTTP endpoint and its auth.
• Training Pipeline — the eight-stage chain, GPU providers, and the quality gate.