Run Lifecycle

A run is one attempt to produce a trained avatar. It is a single ml.app_runs row that carries the request, the review payload, and the artifact manifest, and whose status column is the authoritative state of the whole workflow. This page documents the status vocabulary, the allowed transitions, how jobs attach to a run, and how the review window closes itself.

Statuses

common/statuses.py defines the vocabulary. Every status maps to a coarse stage, which patch_run writes automatically whenever a status changes without an explicit stage.

Status	Stage	Meaning
`created`	`workflow`	Row inserted; domain matching has not been recorded yet.
`domain_confirmation_required`	`workflow`	The embedding match was not confident; the creator must pick a domain.
`seed_examples_pending`	`workflow`	Domain settled, but fewer than `SEED_EXAMPLES_MIN` seed examples were supplied.
`smart_queued`	`workflow`	A `smart_generate` job is queued.
`smart_running`	`workflow`	Kimi generation in progress.
`review_pending`	`workflow`	Samples generated; awaiting the creator's decision.
`review_approved`	`workflow`	Approved, manually or automatically.
`review_rejected`	`workflow`	Rejected. Terminal in practice — `next_action` is `none`.
`postprocess_running`	`workflow`	A `smart_finalize` job is blending the final dataset.
`training_queued`	`training`	A `train` job exists and the chain has been dispatched.
`training_instance_requested`	`training`	A GPU instance/pod has been requested.
`training_instance_ready`	`training`	SSH on the instance answered a `READY` probe.
`training_payload_uploaded`	`training`	Dataset, manifest, metadata, and training script are on the instance.
`training_started`	`training`	The detached remote training process reported its PID.
`training_monitoring`	`training`	Polling the remote PID until it exits.
`training_artifacts_synced`	`training`	The adapter bundle has been downloaded and the avatar profile published.
`completed`	`completed`	Quality gate written, LoRA evaluated, job closed.
`failed`	`failed`	A stage failed terminally.
`cancelled`	`cancelled`	Run abandoned.
`training_ready`	`training`	Legacy status kept so older rows can rejoin the canonical flow.

completed, failed, and cancelled are terminal: ALLOWED_TRANSITIONS maps each to an empty set, and every worker treats a terminal run as a signal to retire the job rather than retry it.

Transitions

patch_run validates the target status and rejects any move not listed in ALLOWED_TRANSITIONS, raising invalid run transition: <from> -> <to>. Re-applying the current status is always allowed, which is what makes the training stages idempotent.

created
  ├─► domain_confirmation_required ─┐
  ├─► seed_examples_pending ────────┤
  └─► smart_queued ◄────────────────┘
        │
        ▼
   smart_running ──► review_pending
                        │
        ┌───────────────┼──────────────────┐
        ▼               ▼                  ▼
  smart_queued    review_approved    review_rejected
 (edit_request)         │                  │
                        ▼                  └─► smart_queued
                postprocess_running
                        │
                        ▼
                 training_queued
                        ▼
          training_instance_requested
                        ▼
             training_instance_ready
                        ▼
            training_payload_uploaded
                        ▼
                 training_started
                        ▼
               training_monitoring
                        ▼
            training_artifacts_synced
                        ▼
                    completed

Every non-terminal status may also move to failed or cancelled.

next_action

Run responses include a next_action string so clients do not have to interpret statuses themselves.

`next_action`	Statuses
`confirm_domain`	`domain_confirmation_required`
`submit_seed_examples`	`seed_examples_pending`
`submit_review`	`review_pending`
`wait_for_training`	`training_queued` … `training_monitoring`
`wait_for_training_completion`	`training_artifacts_synced`
`none`	`completed`, `review_rejected`, `failed`, and anything unmapped

Run creation

WorkflowCore.create_run does the following, in order:

Normalize the ElevenLabs voice id (^[A-Za-z0-9_-]{1,64}$, or null).
Match specialty_text against the domain catalog, or accept domain_id_override verbatim at confidence 1.0.
Insert the app_runs row with status='created' and the full domain decision in request_payload.
Persist the avatar → domain assignment to state/config/avatar_domain_mapping.json and mirror the domain's catalog fields (name, description, vertical tag) onto the avatar row.
Choose the next status: domain_confirmation_required if the match needs confirmation, else seed_examples_pending if there are not enough seeds, else smart_queued.
When the status is smart_queued, create the smart_generate job and enqueue it.
Write the run's summary artifacts and append a run_initialized event.

If anything in steps 2–7 raises, the run is patched to failed, a run_initialization_failed event is appended, and the original exception propagates. Both cleanup calls are themselves wrapped so a secondary failure cannot mask the first one.

Guards on creator actions

Action	Required status	Extra guards
`confirm_domain`	`domain_confirmation_required`	The domain must exist in the catalog and its `{domain_id}_base.json` dataset must be present on disk.
`submit_seed_examples`	`seed_examples_pending`	Count must be within `SEED_EXAMPLES_MIN`–`SEED_EXAMPLES_MAX`; the run must already have a confirmed domain; if a job already exists, an identical resubmission is idempotent and anything else is rejected.
`submit_review`	`review_pending`	The run must have a `current_job_id`.

Every one of these also enforces ownership: when a user_id is supplied and does not match the run's creator_id, the call fails with access denied: run <id> belongs to another user.

Jobs

Jobs live in ml.app_jobs and are always created with status='queued'.

`job_type`	Enqueued by	Celery task	Eligible run statuses
`smart_generate`	Run creation, domain confirmation, seed submission, or an `edit_request` review	`smart.generate`	`smart_queued`, `smart_running`
`smart_finalize`	Review approval (manual or automatic)	`smart.finalize`	`review_approved`, `postprocess_running`
`train`	`smart.finalize`, after the final dataset is written	the eight `training.*` tasks	all seven `training_*` statuses plus legacy `training_ready`

Two invariants keep duplicate work out:

One live training lifecycle per run. create_job takes SELECT … FOR UPDATE on the run row and, if a queued or running train job already exists, returns it with _existing: True and appends a job_reused event instead of inserting. This is what prevents a retried finalize from renting a second GPU.
Only the current job may be claimed. app_runs.current_job_id points at the job that owns the run; a claim query skips any tracked job that is not the current one.

The review window

When smart.generate finishes it stamps a review window into review_payload and schedules three countdown tasks on the smart queue:

Task	Default delay	Effect
`smart.review_reminder_24`	`PERS_REVIEW_REMINDER_1_SEC` (24 h)	Sets `review_window.reminder_24_sent`.
`smart.review_reminder_48`	`PERS_REVIEW_REMINDER_2_SEC` (48 h)	Sets `review_window.reminder_48_sent`.
`smart.finalize_review`	`PERS_REVIEW_FINALIZE_SEC` (72 h)	Auto-approves a still-pending review.

Setting PERS_REVIEW_TIMER_MODE=test changes the defaults to 24 / 48 / 72 seconds, which is what the shipped test profile uses. Explicit PERS_REVIEW_*_SEC values always win.

review_epoch

Each generation pass increments review_payload.review_epoch. All three timer tasks receive the epoch they were scheduled for and return ignored_epoch_mismatch when it no longer matches. This is what makes an edit_request safe: the regenerated run gets a fresh epoch, and the old timers become no-ops instead of auto-approving stale samples.

Reminders additionally return ignored_non_pending once the review status is no longer pending.

Auto-approval

When the deadline fires on a still-pending review, _auto_finalize_review writes a synthetic decision — action: "auto_approve", accepted: true, decision_source: "auto" — moves the run through review_approved to postprocess_running, creates the smart_finalize job with reason review_auto_approved, and appends a review_auto_approved event. If the review had already been approved, it only backfills finalized_at and returns already_approved.

Manual review

POST /v1/runs/{run_id}/review accepts three actions:

Action	Resulting review status	Run transition
`approve`	`approved`	`review_approved` → `postprocess_running`, `smart_finalize` job created and enqueued.
`edit_request`	`edit_request`	Stays `review_pending`; a new `smart_generate` job is created (which then patches the run to `smart_queued`).
`reject`	`rejected`	`review_rejected`. Nothing further is scheduled.

approve and reject also stamp review_window.finalized_at and decision_source: "manual", which stops the auto-approval path even before the epoch check.

Failure and retry behaviour

Celery task wrappers share one shape: classify the error, retry with exponential backoff (countdown = 2 ** retries) while attempts remain, then fail the job and the run.

Task group	Max retries	Retry condition
`smart.generate`	3	Only transient-looking Kimi errors (timeout, 429, 5xx, rate limit, connection).
`smart.finalize`	3	Any error.
`smart.review_reminder_*`, `smart.finalize_review`	2	Any error.
`training.*`	4 (3 for `training.process`)	Any error. Each retry also increments `app_jobs.attempt_no`.

Stale-job errors are never retried. Messages containing invalid run transition, stale job, stale training, terminal run, or cannot start training from run status, and KeyErrors for a missing job or run, mean the world moved on. The job is retired (status='cancelled' with the reason recorded) and the task raises Celery's Ignore or returns ignored_stale_task.

Terminal training failures additionally call _best_effort_terminate_job_instance, so a rented GPU is not left running after the chain gives up. See Training Pipeline.

Job recovery

The optional worker_runtime loop in training/tasks.py runs three repair passes every five seconds:

Pass	Effect
`repair_orphaned_run_jobs`	For a run sitting in a job-bearing status with no active job, re-points `current_job_id` at a suitable active job or recreates the missing job. Appends `current_job_repaired` or `orphaned_run_repaired`.
`retire_ineligible_jobs`	Cancels jobs whose run is missing, terminal, or in a status that job type may not run in, and jobs superseded by a newer `current_job_id`. Default staleness threshold 30 s (`WORKER_RUNTIME_STALE_RETIRE_SECONDS`).
`requeue_stale_running_jobs`	Returns `running` jobs whose `updated_at` has gone quiet to `queued` — 180 s for smart jobs (`WORKER_RUNTIME_STALE_REQUEUE_SECONDS`), at least 1200 s for `train`.

Long-running work keeps itself alive against these thresholds by calling touch_job, which bumps updated_at for queued or running jobs only. Kimi generation touches on every heartbeat window, and the training monitor touches on every poll.

Inline job processing — claiming jobs from Postgres with SELECT … FOR UPDATE SKIP LOCKED instead of consuming from Redis — is separate, and stays off unless WORKER_RUNTIME_INLINE_ENABLED=true. In that mode train jobs are run in daemon threads capped by WORKER_RUNTIME_MAX_PARALLEL_TRAIN_JOBS (default 1), while smart jobs run synchronously in the loop.

Events

Every meaningful step appends a row to ml.app_events with severity='info' and actor_type='service'. The event_type values double as an audit vocabulary:

run_created · run_initialized · run_initialization_failed · domain_confirmed · seed_examples_submitted · job_created · job_reused · job_retired · smart_generation_completed · review_reminder_24 · review_reminder_48 · review_submitted · review_auto_approved · final_dataset_ready · training_queued · training_instance_requested · training_instance_ready · training_payload_uploaded · training_started · training_monitoring · training_artifacts_synced · training_instance_terminated · training_instance_cleanup · training_completed · current_job_repaired · orphaned_run_repaired · worker_heartbeat

Statuses​

Transitions​

next_action​

Run creation​

Guards on creator actions​

Jobs​

The review window​

review_epoch​

Auto-approval​

Manual review​

Failure and retry behaviour​

Job recovery​

Events​