Setup and Operations

How to install Onboarding V4, configure it, start and stop the three processes, and validate the database schema and the domain configuration before serving traffic.

Prerequisites

Dependency	Used for
Python 3.11+	from __future__ import annotations with PEP 604 unions is used throughout; tarfile.extractall(filter="data") requires 3.12 or a 3.11 backport.
PostgreSQL	The single ml schema. pgcrypto is needed for gen_random_uuid(); the initializer skips it silently if the app user lacks the privilege.
Redis	Celery broker (/4) and result backend (/5) in the shipped .env.example.
TEI embeddings server	Domain matching, deduplication, and style scoring. intfloat/multilingual-e5-large, reachable at EMBEDDER_BASE_URL.
OpenRouter account	Kimi K2.5 for dataset generation, prompt rewriting, and LoRA judging.
RunPod and/or Vast.ai account	GPU instances for LoRA training, plus an SSH keypair per provider.
ElevenLabs account	Only for regenerating the static voice previews.
ssh / scp / ssh-keygen	Remote training is driven entirely through the system SSH client.

Install

python -m venv .venv
.venv/bin/pip install -r requirements.txt

requirements.txt pins the API and worker stack (fastapi, uvicorn, pydantic, celery, redis, psycopg[binary], openai, sentence-transformers) plus the training stack (torch, transformers, datasets, peft) that training/scripts/train_unsloth.py needs on the remote GPU instance.

note

unsloth and bitsandbytes are not in requirements.txt. train_unsloth.py imports both defensively: without unsloth it falls back to a plain transformers + peft backend, and without bitsandbytes it uses adamw_torch instead of adamw_8bit. Provide them in the remote GPU image.

Environment

common/config.load_environment() loads .env files with override=False, so already-exported variables always win. It tries, in order:

1. UNIFIED_ENV_FILE, if set;
2. ${WORKSPACE_ROOT}/../.env;
3. ${WORKSPACE_ROOT}/.env.

WORKSPACE_ROOT defaults to the current working directory, and RUNTIME_ROOT defaults to ${WORKSPACE_ROOT}/state.

Core runtime

Variable	Default	Purpose
WORKSPACE_ROOT	cwd	Repository root; also where training/scripts/train_unsloth.py is looked up.
RUNTIME_ROOT	${WORKSPACE_ROOT}/state	Runtime state tree (see Data Model).
LOG_DIR	${RUNTIME_ROOT}/logs	Log root.
LOG_LEVEL / LOG_JSON	INFO / false	Log level; LOG_JSON=true switches to one-line JSON records.
API_HOST / API_PORT	0.0.0.0 / 8091	Uvicorn bind. .env.example and API_HEALTH_URL use 8094.
ONBOARDING_CORS_ORIGINS	empty	Comma-separated CORS allow-list. Empty means no cross-origin browser access.
WS_AGENT_API_KEY	—	Shared secret for the X-Api-Key-guarded avatar endpoints. Required; requests get 503 when unset.
SEED_EXAMPLES_MIN / SEED_EXAMPLES_MAX	20 / 50	Accepted seed-example count. The Pydantic request models hard-code the same 20–50 bounds.

Database

Variable	Default	Purpose
DB_DSN	built from POSTGRES_*	Full connection string. Takes precedence when non-empty.
POSTGRES_USER / _PASSWORD / _HOST / _PORT / _DB	postgres_user / change-me / 127.0.0.1 / 5435 / postgres_db	Used to assemble the DSN when DB_DSN is empty.
DB_SCHEMA	ml	Must be ml. Any other value raises at startup.
DB_SKIP_DDL_INIT	false	When true, startup only asserts the schema is ready instead of running DDL.

Celery and Redis

Variable	Default	Purpose
CELERY_BROKER_URL	redis://127.0.0.1:${REDIS_PORT}/0	Broker. .env.example uses database /4.
CELERY_RESULT_BACKEND	redis://127.0.0.1:${REDIS_PORT}/1	Result backend. .env.example uses /5.
REDIS_PORT	6391	Fallback used to build both defaults. .env.example uses 6356.
CELERY_TASK_ALWAYS_EAGER	false	Runs tasks inline in the calling process. Test-only.

Queue routing is fixed in common/celery_app.py: smart.* tasks go to the smart queue, training.* tasks to training, and anything else to the default workflow queue. Workers run task_acks_late=True with worker_prefetch_multiplier=1 and task_reject_on_worker_lost=True.

Generation, matching, and training

The remaining variables are documented next to the behaviour they control:

• domain matching and the embeddings backend — Domain Matching;
• Kimi generation, style scoring, deduplication, and review timers — Dataset Generation;
• RunPod, Vast.ai, SSH, and adapter storage — Training Pipeline;
• the ElevenLabs voice roster and previews — Voice.

warning

.env.example also ships PERS_GEN_TARGET_TOTAL, PERS_GEN_TARGET_STANDALONE, PERS_GEN_TARGET_RAG, PERS_KIMI_TARGET_TOTAL, PERS_MAX_GENERATION_RETRIES, PERS_ALLOW_MOCK_KIMI, and EMBEDDING_DIMENSION. None of them are read anywhere in the code. Sample counts come from the per-seed quotas instead, and mock Kimi endpoints cannot be enabled at all.

Initialize the schema

Schema creation is idempotent and can be run on its own:

python -m common.schema

It creates the ml schema, takes a transaction-level advisory lock so several starting processes cannot race the DDL, creates the seven tables and their indexes, applies the in-place user_id → creator_id migrations, backfills avatar catalog fields from the domain catalog, and finally asserts that every owned table exists.

Two guards protect against a half-migrated database:

• initialization refuses to build an empty ml schema while the same canonical tables still exist in public;
• assert_schema_ready() reports which tables are missing, and points at leftover public tables when it finds them.

Validate the domain configuration

The catalog, base datasets, LoRA evaluation questions, avatar mapping, and checksum manifest are validated together:

python -m domain.integrity --config-root state/config

Migration mode normalizes the avatar mapping and regenerates checksums before validating:

python -m domain.integrity --config-root state/config --fix

See Domain Matching for what each check covers.

Start, stop, status

./ops/start_all.sh

start_all.sh sources .env, creates the log and PID directories, forces 600 permissions on both provider SSH keys, runs python -m common.schema, then starts the API and the two Celery workers with nohup. It waits up to 30 s for API_HEALTH_URL to answer and up to 30 s for both smart@$(hostname) and training@$(hostname) to answer celery inspect ping. On any failure the ERR trap kills whatever was already started.

./ops/stop_all.sh
./ops/status_all.sh

stop_all.sh sends SIGTERM to each recorded PID and removes the PID file. status_all.sh prints running/stopped per process and exits non-zero if any is down.

Health check

curl -fsS http://127.0.0.1:8094/health

Returns {"status": "ok", "service": "workflow"}. It touches neither the database nor any upstream, so it is a pure liveness probe.

Logs

Path	Written by
state/logs/workers/smart_worker.log	The API process (logger app_workflow) and, via nohup, the smart Celery worker.
state/logs/workers/training_worker.log	The training Celery worker and the optional worker_runtime.
state/logs/training/lora_quality_eval.log	The LoRA quality evaluator (logger lora_quality_eval).
state/logs/ops/{start_all,stop_all,status_all}/	Created by ensure_runtime_dirs() for ops output.

note

The API and the smart worker share smart_worker.log. When tracing a request end to end, filter on the logger name: app_workflow for the API, worker_runtime for the polling worker.