Data Model

Onboarding V4 keeps state in two places: the PostgreSQL ml schema, which is authoritative, and a local state/ tree that mirrors summaries as JSON files and stores the binary artifacts (adapters, audio) that do not belong in a database.

Schema

The schema name is fixed. resolve_db_schema() raises at import time if DB_SCHEMA is anything other than ml, and every connection is opened with search_path=ml.

Table	Owner of
app_runs	The onboarding + training lifecycle. One row per run.
avatars	Final avatar state, including the runtime prompt and adapter path. Read by LIS 3.
app_jobs	Generation, finalization, and training jobs.
app_events	Append-only audit trail.
app_sources, app_source_contents, app_source_chunks	Knowledge ingestion. Created but unused by this codebase.

app_runs

Column	Type	Notes
run_id	text PK	run_ + 12 hex characters.
creator_id	text	Migrated in place from a former user_id column.
avatar_id	uuid
status	text NOT NULL	See Run Lifecycle.
stage	text NOT NULL	Derived from status: workflow, training, completed, failed, cancelled.
domain_id	text	Set once the domain is settled.
request_payload	jsonb	The creator's request plus the domain decision.
review_payload	jsonb	Generated samples, review state, review window, avatar prompt.
artifact_manifest	jsonb	Summaries, final dataset, training manifest, quality results, artifact paths.
current_job_id	text	The job that owns the run right now.
created_at, updated_at, completed_at	timestamptz	completed_at is stamped on completed and failed.

Index: idx_app_runs_status.

avatars

Column	Type	Notes
avatar_id	uuid PK	Supplied by the caller.
uid	uuid NOT NULL	gen_random_uuid() default, uniquely indexed.
creator_id	text
name	text	System name, derived from the domain's display name.
user_avatar_name	text	Creator's override; wins over name in API responses.
description	text	The domain's first description.
tags	jsonb	[vertical], or [].
preview_url	text	Not written by this service.
customization	jsonb	Holds character_story.
collection_name	text	Nullable; empty strings are normalized to NULL.
is_published	boolean	Creator-controlled.
lora_quality_passed	boolean	The evaluation verdict. GET /v1/avatars?trained_only=true filters on it.
lora_quality_score	double precision	Average judged accuracy.
lora_pass_report	text	Path to lora_quality_report.json.
lora_adapter_path	text	Local adapter directory.
system_prompt	text NOT NULL, default ''	The refined runtime prompt.
voice_clone_path, voice_clone_text	text	Uploaded reference audio and its script.
elevenlabs_voice_id	text	Selected roster voice.
status	text NOT NULL, default active	active or deleted.
date, updated_at	timestamptz	date is the creation timestamp.

Indexes: idx_avatars_uid_unique, idx_avatars_creator on (creator_id, updated_at DESC), and a partial idx_avatars_active_catalog on (updated_at DESC, avatar_id) WHERE status = 'active'.

upsert_avatar is field-preserving: NULL or empty-string inputs keep the stored value instead of clearing it, [] keeps existing tags, {} keeps existing customization, and the ON CONFLICT branch carries WHERE avatars.status = 'active' so a soft-deleted avatar is never silently resurrected.

app_jobs

Column	Type	Notes
job_id	text PK	job_ + 12 hex characters.
run_id	text FK → app_runs	ON DELETE SET NULL.
source_id	text FK → app_sources	Unused.
job_type	text NOT NULL	smart_generate, smart_finalize, train.
status	text NOT NULL	queued, running, completed, failed, cancelled.
attempt_no	integer NOT NULL, default 1	Incremented on every task retry.
payload	jsonb	Job input; training jobs also carry the training manifest.
error_message	text	Failure or retirement reason.
created_at, updated_at, finished_at	timestamptz	updated_at doubles as the liveness heartbeat.

Indexes: idx_app_jobs_status, idx_app_jobs_claim on (status, job_type, created_at ASC), idx_app_jobs_run_active on (run_id, status, job_type).

Eligibility is enforced in SQL, not only in Python. common/db_utils.py maps each job type to the run statuses it may run in, and both the claim query and the maintenance sweeps apply that map plus a current_job_id guard:

job_type	Eligible run statuses
smart_generate	smart_queued, smart_running
smart_finalize	review_approved, postprocess_running
train	the seven training_* statuses plus legacy training_ready

app_events

Column	Type	Notes
event_id	bigserial PK
run_id, source_id, job_id	text FK	All ON DELETE SET NULL.
event_type	text NOT NULL	See the event vocabulary.
severity, actor_type, actor_id	text	Currently always info / service / unset.
message	text	Human-readable summary.
payload	jsonb	Full context — often the whole training manifest.
created_at	timestamptz

Index: idx_app_events_run on (run_id, created_at DESC).

Events are also a fallback data source: instance cleanup reconstructs provider state from the most recent training_started / training_payload_uploaded / training_instance_ready / training_instance_requested payload when both the run manifest and the job payload have gone stale.

JSONB writes

All JSONB goes through _jsonb(), which strips \x00 from every string, converts sets and tuples to lists, and serializes anything else with default=str — so a datetime or UUID never breaks a write.

Two merge helpers do shallow top-level merges with the || operator:

• merge_run_artifacts(run_id, patch) — artifact_manifest = artifact_manifest || patch;
• merge_job_payload(job_id, patch) — payload = payload || patch.

Because the merge is shallow, a patch replaces a whole top-level key rather than merging into it. This is why the training stages rebuild the manifest and copy forward runpod, vast, payload, command, monitor, and artifacts explicitly.

JSONB payload shapes

request_payload

{
  "mode": "train",
  "specialty_text": "…",
  "seed_examples": ["…"],
  "elevenlabs_voice_id": "JBFqnCBsd6RMkjVDRZzb",
  "domain_override": null,
  "domain_decision": {
    "domain_id": "math",
    "confidence": 0.91,
    "needs_confirmation": false,
    "top_matches": [ { "domain_id": "math", "domain": "…", "score": 0.91 } ],
    "threshold_used": 0.85,
    "margin": 0.07,
    "margin_threshold_used": 0.01,
    "normalized_input": "…",
    "dataset_file": "…/math_base.json",
    "sample_count": 41
  }
}

review_payload

{
  "generated_samples": [ { "instruction": "…", "output": "…", "type": "standalone", "personalization": { … } } ],
  "review_status": "pending",
  "generated_at": "2026-06-29T11:45:23+00:00",
  "generation_source": "kimi",
  "avatar_prompt": {
    "domain_id": "math",
    "base_system_prompt": "…",
    "refined_system_prompt": "…",
    "cleaned_seed_examples": ["…"],
    "status": "published",
    "error": null
  },
  "domain_decision": { … },
  "domain_confirmation": { … },
  "review_epoch": 1,
  "review_window": {
    "started_at": "…",
    "deadline_at": "…",
    "scheduled_in_seconds": { "reminder_24": 86400, "reminder_48": 172800, "finalize": 259200 },
    "reminder_24_sent": false,
    "reminder_48_sent": false,
    "finalized_at": null,
    "decision_source": null
  },
  "review": {
    "action": "approve",
    "note": "",
    "accepted": true,
    "needs_regeneration": false,
    "reviewed_at": "…",
    "decision_source": "manual"
  }
}

review_status takes pending, queued, approved, auto_approved, rejected, or edit_request. review.action is approve, edit_request, reject, or auto_approve.

artifact_manifest

Key	Written by
smart, review, generated	Every artifact sync — generation and review summaries.
final_dataset	smart.finalize — the blended dataset and its metadata.
final_dataset_summary	Derived summary (path, counts, blend metadata).
training	Every training stage — the full manifest.
training_job_id, training_job_status	Convenience mirrors of the training job's id and pipeline status.
quality_gate	training.complete — the handoff checklist.
lora_quality	training.complete — the evaluation verdict and report path.
avatar_profile	training.sync_artifacts — adapter publication state.
artifact_paths	Absolute paths of every JSON file written under state/runs/{run_id}/.

The training manifest carries provider state across stages:

{
  "job_id": "job_…", "run_id": "run_…", "avatar_id": "…", "domain_id": "math",
  "pipeline_status": "training_monitoring",
  "canonical_run_status": "training_monitoring",
  "provider": "runpod",
  "dataset": { "source": "artifact_manifest.final_dataset.examples", "example_count": 236, "ready_for_training": true },
  "runpod": { "enabled": true, "status": "ssh_ready", "pod_id": "…", "ssh_host": "…", "ssh_port": 12345 },
  "vast": { "enabled": true, "status": "fallback_not_used" },
  "payload": { "remote_dataset_path": "…", "remote_output_dir": "…", "remote_log_path": "…" },
  "command": { "command": "python …", "remote_train_pid": "1234", "started_at": "…" },
  "monitor": { "status": "training_succeeded", "final_adapter_remote_path": "…", "verified_outputs": [ … ] },
  "artifacts": { "artifact_dir": "…", "adapter_dir": "…", "files": [ … ] }
}

Runtime state

ensure_runtime_dirs() creates the tree on every process start.

state/
├── config/
│   ├── domains.json                      # 26-domain catalog
│   ├── domain_datasets/{domain_id}_base.json
│   ├── lora_eval_questions/{domain_id}.json
│   ├── avatar_domain_mapping.json        # mutable; excluded from checksums
│   ├── bootstrap_checksums.json          # SHA-256 manifest
│   └── domain_candidates/{domain_id}/    # unreleased packs
├── runs/{run_id}/
│   ├── smart_job.json
│   ├── review.json
│   ├── generated.json
│   ├── final_dataset.json
│   ├── final_dataset_summary.json
│   ├── training_job.json
│   ├── avatar_profile.json
│   └── quality_gate.json
├── models/{avatar_id}/
│   ├── adapter_config.json
│   ├── adapter_model.safetensors
│   ├── tokenizer*.json, vocab.json, merges.txt, …
│   ├── training_summary.json
│   ├── training_metadata.json
│   ├── lora_quality_report.json
│   └── train.log                          # only with TRAINING_DOWNLOAD_REMOTE_LOG=true
├── voice/
│   ├── previews/manifest.json
│   ├── previews/audio/{key}.mp3
│   └── clones/{creator_id}/{avatar_id}/voice_clone_*.{wav,mp3,m4a,ogg,webm}
├── secrets/{runpod,vast}/id_ed25519      # chmod 600, enforced on every use
├── logs/
│   ├── workers/{smart_worker,training_worker}.log
│   ├── training/lora_quality_eval.log
│   └── ops/{start_all,stop_all,status_all}/
└── pids/{api,smart_worker,training_worker}.pid

note

Files under state/runs/ are mirrors, not sources of truth — everything in them also lives in artifact_manifest. Files under state/models/ are not mirrored: an adapter directory is the only copy, and training.sync_artifacts deletes and recreates it on each run, so re-running training for the same avatar replaces the previous adapter.

state/models/{avatar_id} uses the normalized avatar id — lowercased, with anything outside [a-z0-9_-] collapsed into single underscores. Standard UUIDs pass through unchanged.