Scenario → Render Pipeline

How the website generates a scenario and what exactly arrives at the render worker. This is the hand-off document for the Unreal Engine / render-worker developer.

note

Derived from frontend/src/Profile/pages/StudioPage/*, frontend/src/api/render.ts, backend/src/routes/render.js, and backend/src/services/avatars/avatarCustomizationService.js. Current as of 2026-07-10.

1. Responsibility boundary

• Web (us) — the wizard UI, generating the scenario text with the selected avatar (through LIS), assembling the payload, proxying to the worker, polling status, and showing the video.
• render-worker + Unreal Engine (you) — accept POST /render, parse the Scenario DSL (emotions + on-screen captions), apply Customization, synthesize speech, render the video, and return a signed URL.

All scenario generation happens on the web side. The worker receives a finished Scenario string with inline markers; your job is to parse and render it.

2. End-to-end process

The Video Creator wizard (/app/studio/create) runs Theme → Scenario → Render:

1. Visual / avatar choice. The user picks an avatar; we take the real avatarId from ws-agent. It travels in the payload, and you resolve the model from it.
2. Theme. A form with Name, Theme, Description, Format (16:9 / 9:16), Resolution (FullHD / HD), and Duration (15 s / 30 s / 1 min).
3. Scenario (text generation). The Generate Scenario button:
   • creates a temporary LIS session with the selected avatarId (the same platform as chat, so the avatar's persona and style are applied automatically from avatar_id);
   • sends one prompt: write a first-person monologue scenario on the topic …, target duration ≈ N seconds, place emotional tags, duplicate key words as on-screen captions;
   • collects the streamed response and splits it on blank lines into "cells" (a Google-Colab-like model: each cell can be edited or regenerated on its own);
   • deletes the temporary session, so the user's chat list stays clean.
   • The user can edit cells by hand, regenerate a single one (▶), or add a new one.
4. Combine. The cells are joined back into a single Scenario string, preserving the blank line between cells.
5. Render. Before sending, we fetch the avatar's customization from our backend (GET /api/avatars/:id/customization) and put it in the payload. If no customization exists yet (404), the render does not start and the user is told to set the avatar customization first. Then POST /render → poll GET /render/:id every 3 s → on completed, show videoUrl.

[Avatar pick] → [Theme form] → [LIS generates scenario → cells] → [Combine]
      │                                                               │
      └──────────────► GET /api/avatars/:id/customization ◄───────────┘
                                     │
                                     ▼
                    POST /render  (AvatarId + Render + Customization)
                                     │  (202: { id })
                                     ▼
                    GET /render/:id  ──(until completed/failed)──┐
                                     │                            │
                                     ▼                            │
                          videoUrl (signed) ◄────────────────────┘

3. Scenario DSL

This is the most important part for the worker's parser. The Render.Scenario string contains plain spoken text plus two kinds of inline marker.

3.1 Emotion tags

Placed at the start of a line. Only these five are allowed (agreed 2026-06-25):

{<HAPPY>}  {<ANGRY>}  {<CONCERNED>}  {<STARE>}  {<NEUTRAL>}

The emotion applies to the line it precedes. The model emits no other tags.

3.2 On-screen caption (SHTEXT overlay)

Duplicates a key word as text on screen. Placed inline, immediately after the word. The marker format is exact:

[<{<SHTEXT: short text>}, {<ShowTime: 4>}>]

• SHTEXT — the short caption text (1–3 words, not a whole sentence).
• ShowTime — how many seconds the caption stays on screen (typically 4, range 4–6).
• The position is fixed — to the right of the avatar (the render server default; there is no separate position tag — confirmed 2026-06-29).
• The caption adds nothing to the speech — the word is already spoken in the sentence, and the marker only duplicates it visually. Frequency is roughly one caption per 2–3 sentences.

3.3 Structure

Lines are short (1–2 sentences) and separated by a blank line — those are the cell boundaries.

3.4 Example of a finished Scenario

{<HAPPY>} Hi! Today I'll show how our render pipeline works [<{<SHTEXT: render pipeline>}, {<ShowTime: 4>}>].

{<NEUTRAL>} First you pick an avatar and set the video topic.

{<CONCERNED>} If you skip the customization step [<{<SHTEXT: customization>}, {<ShowTime: 5>}>], the render will not start.

4. render-worker API

Base URL: https://render.amadeq.download (development http://…:3100).

The web app does not call it directly but through a same-origin proxy at /api/render on our backend, because the worker has no CORS: the backend transparently forwards Authorization and the body to {base}/render…. That is invisible to you — the paths below are the worker's own.

Authorization

• Bearer JWT in Authorization: Bearer <accessToken>.
• The token is the same access JWT the site issues; the JWT_SECRET is shared.
• userId comes from the userId claim, not from sub.

warning

A top-level UserId in the body is ignored — the job owner is determined solely from the JWT. Historically this caused a "test video" to be rendered when the body was trusted instead.

Endpoints

Method	Path	Description
GET	/healthz	Health check. Public, no JWT.
POST	/render	Enqueue a job. 202 { id, status: "queued", statusUrl }. Body — see §5.
GET	/render	List your own jobs (newest first, ≤ 50). snake_case; video_url is unsigned.
GET	/render/:id	Status of one job. camelCase; videoUrl is signed, TTL ~1 hour, present only when completed.
DELETE	/render/:id	Delete the job and its files on disk. Irreversible. Returns { id, deleted: true }; 404 for someone else's or a missing job.

Statuses

queued → processing → completed | failed.

Response shapes

// POST /render → 202
{ "id": "…", "status": "queued", "statusUrl": "…" }

// GET /render/:id  (camelCase, videoUrl signed when completed)
{ "id":"…", "status":"completed", "videoUrl":"https://…?sig=…",
  "error": null, "createdAt":"…", "finishedAt":"…" }

// GET /render  (list; snake_case, video_url unsigned)
{ "jobs": [ { "id":"…", "status":"…", "video_url":null,
             "created_at":"…", "finished_at":null } ] }

5. POST /render body (RenderJobInput)

The numeric Render.* fields are preset indices — the value-to-appearance mapping is defined by UE — except Duration, which is in seconds.

{
  "AvatarId": "34d4bd1c-…",          // string 1–100; resolves the model
  "Render": {
    "Name":        "Intro video",    // 1–200
    "Theme":       "Onboarding",     // ≤100
    "Description": "Short intro",    // ≤2000
    "Format":      0,                // 0 = 16:9, 1 = 9:16
    "Resolution":  0,                // 0 = FullHD, 1 = HD
    "Duration":    15,               // seconds, 1–600 (15 / 30 / 60)
    "Scenario":    "{<HAPPY>} Hi!…"  // ≤20000, the DSL from §3
  },
  "Customization": { /* §6 — the whole object is required */ }
}

UI mapping, for reference: Format 16:9→0, 9:16→1; Resolution FullHD→0, HD→1; Duration "15 sec"→15, "30 sec"→30, "1 min"→60.

6. Customization (avatar appearance)

The whole object is required — the worker's schema demands every branch. Numbers may be fractional: some fields (for example Head.Nose/Mouse/Cheeks/Ears) are continuous morph values such as 1.1, not integer indices.

{
  "Body":  { "Name":"", "Gender":0, "Age":0, "Type":0, "Texture":0 },   // Gender 0–2, Age 0–120
  "Head":  { "Hair":0,"HairColor":0,"Beard":0,"BeardColor":0,
             "Eyelashes":0,"Eyebrows":0,"EyebrowsColor":0,
             "Mustache":0,"MustacheColor":0,
             "Nose":0,"Mouse":0,"Cheeks":0,"Ears":0 },                   // these 4 are float
  "Cloth": { "Type":0,"Presets":0,"Outerwear":0,"Underwear":0,
             "Hats":0,"Glasses":0,"Footwear":0 },
  "Study": { "FastPresets":0, "Gender":0, "VoiceType":0 },               // VoiceType 0–50
  "Temperament": 0                                                       // 0–50
}

warning

Known discrepancy: the worker's documentation marks the Head fields as int 0–100, but the deployed worker (after the fix) accepts float for Nose, Mouse, Cheeks, and Ears. Our backend validates them as z.number().finite() — integers and fractions alike. Keep them float.

Where Customization comes from

The customization is stored by our backend, but populated by Unreal:

Method	Path (our backend)	Called by	Purpose
POST	/api/avatars/:id/customization 🔒	Unreal (on the user's behalf)	Ingest/upsert of the avatar's appearance. Accepts { "Customization": {…} } or the bare object. Upsert by avatarId — a repeat push overwrites.
GET	/api/avatars/:id/customization 🔒	Web (the Render step)	Reads what was stored and puts it into the POST /render payload. 404 if it has not been set yet.

• 🔒 = the same Bearer JWT is required.
• Both respond with { "AvatarId", "Customization": {…}, "updatedAt" }.
• Consequence for you: if POST …/customization is never called for an avatar, the web app will not let it be rendered — it fails on the 404 before POST /render. So Unreal pushes the customization first, and it then comes back to you inside the render job.

7. Errors and edge cases

• 404 on GET customization → the web app does not send the render job and shows "set the customization first".
• 401 from the worker → a wrong or expired JWT, or a JWT_SECRET mismatch between the site and the worker. Changing the secret requires users to sign in again, since the old token was signed with the old secret.
• video_url in the GET /render list is unsigned — for playback always take a fresh signed videoUrl from GET /render/:id.
• DELETE is irreversible — it also erases the files from disk.

8. Key files on our side

Concern	File
Scenario generation	frontend/src/Profile/pages/StudioPage/scenarioGen.ts
DSL constants (emotions, SHTEXT)	frontend/src/Profile/pages/StudioPage/videoData.ts
Payload construction and polling	…/components/RenderStep/RenderStep.tsx
Client and contract types	frontend/src/api/render.ts
Proxy to the worker	backend/src/routes/render.js
Customization store	backend/src/services/avatars/avatarCustomizationService.js, routes in backend/src/routes/avatars.js