Dataset Generation

Two Celery jobs turn a creator's seed examples into a training-ready dataset. smart.generate asks Kimi K2.5 for personalized examples and rewrites the avatar's runtime system prompt. smart.finalize — which only runs after review approval — blends those samples with the domain pack into one deduplicated final dataset. Both live in onboarding/generation/.

smart.generate

PersonalizationCore.process_smart_job runs the whole pass:

Mark the job running and the run smart_running.
Validate the live Kimi configuration, then generate examples per seed.
Build the avatar's refined runtime system prompt.
Upsert the avatar row with that prompt and the chosen ElevenLabs voice, resetting lora_quality_passed to false and clearing lora_adapter_path.
Write the samples, generation_source, and avatar_prompt into review_payload; open a fresh review window and bump review_epoch.
Move the run to review_pending, sync artifacts, complete the job, and append smart_generation_completed.
Schedule the 24 h / 48 h reminders and the 72 h auto-finalize task.

Kimi is mandatory

_validate_live_kimi_config runs before anything else and rejects non-production configurations outright:

OPENROUTER_API_KEY (or legacy GROQ_API_KEY), OPENROUTER_BASE_URL (or GROQ_BASE_URL), and PERS_OPENROUTER_MODEL (or PERS_GROQ_MODEL) must all be set;
the base URL must be an absolute URL whose host contains openrouter.ai;
localhost, 127.0.0.1, 0.0.0.0, testserver, mock, example.invalid, and any .local host are refused;
the API key must be at least 20 characters and must not contain test, dummy, fake, mock, changeme, or example.

There is no dataset-only fallback. If Kimi returns fewer samples than the target, the job fails with an explicit "Fallback is disabled in Kimi-only mode" error.

Per-seed quotas

Each seed is one independent request that must produce exactly PERS_PER_SEED_STANDALONE standalone and PERS_PER_SEED_RAG RAG-aware examples — 4 and 1 by default. The target for the run is therefore:

target_total = len(non_empty_seeds) × (PERS_PER_SEED_STANDALONE + PERS_PER_SEED_RAG)

With the default quotas and 20 seeds that is 100 samples. Seeds are capped at 50 by both the request schema and the generator.

Requests run in a ThreadPoolExecutor sized min(len(seeds), PERS_KIMI_MAX_WORKERS). Prompt inputs are clipped before sending: seeds to PERS_SEED_MAX_CHARS, the specialty text to PERS_SPECIALTY_MAX_CHARS, and the domain's base prompt to PERS_BASE_PROMPT_MAX_CHARS.

Retries keep partial progress

Retries are per seed and only ask for the missing remainder. Progress is tracked as a (standalone, rag) pair per seed index; a seed leaves the pending set only once both quotas are met. Between attempts the loop sleeps min(10, 2 ** attempt) seconds, up to PERS_KIMI_MAX_ATTEMPTS attempts.

While waiting on futures, the executor loop wakes every PERS_JOB_HEARTBEAT_SECONDS (20) to call touch_job, which keeps a long generation from being requeued as stale.

After the retries:

if seeds are still unresolved and PERS_STRICT_PER_SEED_QUOTA=true, the job fails and names up to three unresolved seeds with their last error;
otherwise a top-up pass clones existing rows — re-sanitized and tagged source: "kimi_topup" — until both global totals are met. Clones are drawn from the matching type when available, and from the other type when not.

If the top-up cannot reach exactly target_total, the job fails.

Wire format

The model is asked for plain-text blocks, not JSON:

STANDALONE
instruction: ...
input: ...
output: ...
---
RAG
instruction: ...
context: ...
output: ...
rag_behavior: direct_answer|context_synthesis|context_insufficient|multi_source
---

The system prompt forbids JSON, markdown, and explanations, and instructs the model to preserve the creator's style from the seed, stay concise and factual, never copy seed lines verbatim, avoid duplicates, and avoid AI/meta disclaimers. kimi-k2.5 models are called with temperature=0.6, top_p=0.95, and extra_body: {"reasoning": {"effort": "none"}}; other models read PERS_OPENROUTER_TEMPERATURE / PERS_OPENROUTER_TOP_P (defaults 0.6 / 0.9).

_parse_structured_blocks strips code fences, then splits on STANDALONE / RAG headers with --- separators. Field lines are matched case-insensitively and continuation lines are folded into the current field.

_sanitize_generated_example normalizes every row:

whitespace is collapsed in all text fields;
a row without both instruction and output is dropped;
type is forced to standalone or rag_aware;
for rag_aware, rag_behavior outside the four allowed values falls back to context_synthesis, and an empty context is replaced with a placeholder line.

Each kept row carries a personalization block recording source, dataset_file, specialty_text, seed_index, and selection_index.

The avatar runtime prompt

In parallel with the samples, _build_avatar_runtime_prompt_safe asks the model for one production-ready runtime system prompt, given the domain's base_system_prompt and the cleaned seed examples. It is told to keep the domain expertise while adapting tone, teaching style, phrasing patterns, and practical focus, to define behaviour, boundaries, grounding expectations, clarification behaviour, and answer style, and never to mention seeds, metadata, or prompt engineering.

The response is stripped of code fences and a leading System prompt: label, and collapsed to at most one blank line between paragraphs.

This step is deliberately fault-tolerant: on any failure it returns the domain's base prompt with status: "fallback_base_prompt" and the error text, so generation still completes. The result is stored under review_payload.avatar_prompt with status either published or fallback_base_prompt.

smart.finalize

PersonalizationCore.finalize_for_training builds the dataset that will actually be trained.

Preconditions — each raises rather than silently skipping:

the run status must be review_approved or postprocess_running (a terminal status is reported as a stale finalize job);
generated_samples must be non-empty;
review.accepted must be true.

Then:

Load the domain pack and flatten it into examples tagged source: "domain_pack".
Score style — embed both sets, average each into a centroid, and take the cosine similarity between the two centroids.
Categorize the style and pick an oversampling multiplier.
Oversample the personal examples with a random.Random(job_id)-seeded sampler, preserving the standalone/RAG ratio and shuffling the result. Seeding on the job id makes the blend reproducible for a given job.
Concatenate domain examples + oversampled personal examples.
Deduplicate the combined list.
Write the result to artifact_manifest.final_dataset, sync artifacts, complete the job, and append final_dataset_ready.

Style categories

Centroid similarity	Category	Oversampling
`>= PERS_STYLE_CLOSE_THRESHOLD` (0.88)	`close`	1.5×
`>= PERS_STYLE_MODERATE_THRESHOLD` (0.80)	`moderate`	2.0×
below both	`highly_distinct`	3.0×

The more the creator's voice differs from the generic domain pack, the more heavily their own examples are weighted in the mix.

Deduplication

_dedupe_examples_with_min_keep runs a greedy embedding pass: examples are embedded as instruction + input + context + output, and each candidate is dropped when its cosine similarity to any already-kept example reaches PERS_DEDUP_THRESHOLD (0.985).

Because an aggressive pass could gut the dataset, a floor is enforced at len(combined) × PERS_DEDUP_MIN_KEEP_RATIO (0.95). The strategy actually used is recorded in the final metadata:

`dedup_strategy`	When
`embedding_threshold`	The embedding pass kept at least the floor.
`exact_fallback_min_keep_guard`	It did not, so only exact `instruction
`exact_with_topup_min_keep_guard`	Even exact dedup fell below the floor, so rows were cycled back in to reach it.

Final dataset shape

{
  "job_id": "job_…",
  "run_id": "run_…",
  "creator_id": "…",
  "avatar_id": "…",
  "domain_id": "math",
  "examples": [ … ],
  "metadata": {
    "source": "review_payload.generated_samples",
    "review_status": "approved",
    "ready_for_training": true,
    "domain_pack_size": 41,
    "personal_generated_size": 100,
    "oversampling_multiplier": 2.0,
    "stylistic_score": 0.83,
    "stylistic_category": "moderate",
    "combined_before_dedup": 241,
    "combined_after_dedup": 236,
    "removed_by_dedup": 5,
    "dedup_strategy": "embedding_threshold",
    "dedup_threshold": 0.985,
    "dedup_min_keep_ratio": 0.95,
    "domain_dataset_file": "…/math_base.json",
    "generation_provider": "OpenRouter",
    "generation_model": "moonshotai/kimi-k2.5",
    "example_count": 236
  }
}

artifact_manifest.final_dataset.examples is the only accepted training input: _resolve_training_examples raises precondition failed: final_dataset missing when it is absent, so training can never start on unblended samples.

Immediately after finalization, smart.finalize creates the train job and dispatches the training chain. See Training Pipeline.

Configuration

Variable	Default	Purpose
`OPENROUTER_API_KEY` / `GROQ_API_KEY`	—	OpenRouter credential. Validated against mock/test patterns.
`OPENROUTER_BASE_URL` / `GROQ_BASE_URL`	—	Must be an `openrouter.ai` URL. A trailing `/chat/completions` is stripped.
`PERS_OPENROUTER_MODEL` / `PERS_GROQ_MODEL`	—	`moonshotai/kimi-k2.5` in the shipped profile.
`OPENROUTER_TIMEOUT_SEC` / `GROQ_TIMEOUT_SEC`	`120`	Per-request timeout.
`OPENROUTER_HTTP_REFERER`, `OPENROUTER_TITLE`	unset	Optional OpenRouter attribution headers.
`PERS_PER_SEED_STANDALONE`	`4` (max 8)	Standalone examples per seed.
`PERS_PER_SEED_RAG`	`1` (max 4)	RAG-aware examples per seed.
`PERS_KIMI_MAX_ATTEMPTS`	`3` (max 8)	Seed-level retry rounds.
`PERS_KIMI_MAX_WORKERS`	`6` (max 16)	Concurrent seed requests.
`PERS_STRICT_PER_SEED_QUOTA`	`false`	Fail instead of topping up when a seed stays short.
`PERS_SEED_MAX_CHARS`	`260`	Seed clipping. `.env.example` uses 220.
`PERS_SPECIALTY_MAX_CHARS`	`220`	Specialty clipping. `.env.example` uses 180.
`PERS_BASE_PROMPT_MAX_CHARS`	`650`	Base-prompt clipping. `.env.example` uses 550.
`PERS_JOB_HEARTBEAT_SECONDS`	`20`	How often a running generation touches its job.
`PERS_STYLE_CLOSE_THRESHOLD`	`0.88`	`close` style boundary.
`PERS_STYLE_MODERATE_THRESHOLD`	`0.80`	`moderate` style boundary.
`PERS_DEDUP_THRESHOLD`	`0.985`	Cosine similarity at which a duplicate is dropped.
`PERS_DEDUP_MIN_KEEP_RATIO`	`0.95`	Floor on how much of the combined set survives dedup.
`PERS_REVIEW_TIMER_MODE`	`prod`	`test` switches the review-window defaults to seconds.
`PERS_REVIEW_REMINDER_1_SEC` / `_2_SEC` / `PERS_REVIEW_FINALIZE_SEC`	24 h / 48 h / 72 h	Review-window schedule — see Run Lifecycle.

smart.generate​

Kimi is mandatory​

Per-seed quotas​

Retries keep partial progress​

Wire format​

The avatar runtime prompt​

smart.finalize​

Style categories​

Deduplication​

Final dataset shape​

Configuration​