Image Analysis Service

The Image Analysis Service is a stateless FastAPI adapter (default port 8012) that performs question-agnostic visual-context extraction. It validates an uploaded image, then forwards it to a separately-served vLLM multimodal backend and returns a bounded text description. It is consumed internally by the conversation engine and is not exposed through the gateway.

Purpose and topology

The service is a thin, two-tier setup:

• Adapter (:8012) — this service. Validates the image, builds a fixed prompt, calls the backend, and shapes the response. It binds no port itself; the port is supplied by the launcher (see Configuration).
• vLLM backend (:8013) — a separate process serving LiquidAI/LFM2.5-VL-450M over an OpenAI-compatible API. The adapter reaches it at vlm_base_url (default http://localhost:8013).

The service holds no state and touches no database. The visual_text it returns is persisted downstream by the conversation engine into lis.conversation_attachments, not here. For how image turns flow through the system, see Conversation Pipeline.

The prompt is fixed and question-agnostic: no user text is accepted or forwarded. The prompt also instructs the model to treat any text visible inside the image as data, not as instructions to follow (a prompt-injection mitigation), and not to answer or advise the user.

Endpoints

Both routes are mounted at the root — there is no router prefix, and the service is not routed through the gateway's /api_lis prefix.

Method	Path	Description
GET	/health	Readiness probe.
POST	/analyze	Analyze a single image.

GET /health

Returns HTTP 200 when the backend warmup succeeded (ready=true), otherwise HTTP 503.

{
  "service": "image_analysis_service",
  "status": "ok",
  "ready": true,
  "model_id": "LiquidAI/LFM2.5-VL-450M",
  "actual_device": "cuda",
  "warmup_ok": true,
  "startup_error": null
}

Field	Meaning
status	"ok" when ready, else "degraded".
ready	Whether the backend warmup succeeded. Gates /analyze.
model_id	Echoes the configured model_id (not read from the backend).
actual_device	The configured device string — not a probed device.
warmup_ok	Whether the startup warmup inference succeeded.
startup_error	Error string from startup, or null on success.

POST /analyze

Accepts an image only. Extra request fields are rejected.

Request body (AnalyzeRequest):

{
  "image": {
    "mime_type": "image/png",
    "data_base64": "iVBORw0KGgoAAAANSUhEUgAA...",
    "filename": "diagram.png"
  }
}

Field	Type	Required	Notes
image.mime_type	string	yes	Must be in the MIME allowlist.
image.data_base64	string	yes	Strict base64 (validated).
image.filename	string	no	Optional; not used for validation.

Response body (AnalyzeResponse):

{
  "visual_text": "A line chart with three labelled series...",
  "latency_ms": 842.17,
  "model_id": "LiquidAI/LFM2.5-VL-450M",
  "request_latency_ms": 842.17
}

Field	Notes
visual_text	Model output, stripped of surrounding whitespace.
latency_ms	Measured only around the backend call.
model_id	Echoes the configured model_id.
request_latency_ms	Set equal to latency_ms.

note

The response never includes analysis_type from this service. The field exists on the model but stays None and is stripped (response_model_exclude_none=True). Any analysis_type seen by clients originates in the conversation engine.

Error responses:

Status	Error	Cause
400	InvalidImageError	Image failed validation. Fix the input and retry.
503	ModelNotReadyError	Backend not warmed up / not ready.
503	UpstreamError	Backend request failed or returned an empty response.

Both 503 conditions are transient and retryable from the caller's perspective.

Request handling order

/analyze processes a request in a strict order:

1. Validate the image (fail-closed; any failure → 400).
2. Readiness check — if the backend is not ready → 503 (ModelNotReadyError).
3. Build the fixed prompt (question-agnostic).
4. Call the backend — HTTP/backend errors are wrapped as UpstreamError → 503.
5. Reject empty output — an empty or whitespace-only response → UpstreamError → 503.

Validation

validate_image is strictly ordered and fail-closed. Every failure raises InvalidImageError (400):

1. MIME allowlist — mime_type is lowercased and must be in allowed_mime_types.
2. Strict base64 decode — data_base64 is decoded with validation enabled.
3. Size cap — decoded bytes must not exceed max_image_bytes.
4. Magic-byte prefix — the decoded bytes must start with the expected signature for the claimed MIME type.
5. Pillow decode + verify — the bytes are opened and verified with Pillow.
6. Format match — Pillow's detected format must equal the format expected for the claimed MIME type.

Claimed MIME	Magic-byte prefix	Expected format
image/jpeg, image/jpg	ff d8 ff	JPEG
image/png	89 PNG	PNG
image/webp	RIFF	WEBP

Startup and readiness

Startup is fail-soft. On boot, the backend client (VLMClient.startup()):

1. Issues GET {vlm_base_url}/health and raises for non-success status.
2. Runs a warmup inference (max_new_tokens=8) against the backend.

On success, warmup_ok=true, ready=true, startup_error=null. On any exception it sets ready=false and warmup_ok=false, records startup_error, logs a warning, and does not raise — the app still boots and serves. While ready is false, /health returns 503 and /analyze raises ModelNotReadyError.

The warmup image is loaded from warmup_image_path (MIME inferred from the file extension); if the file cannot be read, an embedded 1x1 PNG is used instead.

Backend request shape

The adapter calls the vLLM backend's OpenAI-compatible endpoint:

POST {vlm_base_url}/v1/chat/completions

{
  "model": "LiquidAI/LFM2.5-VL-450M",
  "max_tokens": 256,
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "image_url", "image_url": { "url": "data:image/png;base64,..." } },
        { "type": "text", "text": "<fixed question-agnostic prompt>" }
      ]
    }
  ]
}

The result is taken from choices[0].message.content and stripped. max_tokens is max_new_tokens for /analyze (warmup uses a hardcoded 8).

Configuration

Settings come from ImageAnalysisConfig in the project-root config.py (pydantic-settings; .env.shared / .env, case-insensitive). Most keys accept both an IMAGE_ANALYSIS_* name and a VLM_* alias. The singleton is IMAGE_ANALYSIS_CONFIG (with a backward-compatible VLM_CONFIG alias).

Setting	Env (aliases)	Default	Purpose
model_id	IMAGE_ANALYSIS_MODEL_ID / VLM_MODEL_ID	LiquidAI/LFM2.5-VL-450M	Model name sent to the backend; echoed in responses and /health.
device	IMAGE_ANALYSIS_DEVICE / VLM_DEVICE	cuda	Reported as actual_device only.
vlm_base_url	VLM_BASE_URL / IMAGE_ANALYSIS_VLM_BASE_URL	http://localhost:8013	Base URL of the vLLM backend.
vlm_timeout_sec	VLM_BACKEND_TIMEOUT_SEC / IMAGE_ANALYSIS_VLM_TIMEOUT_SEC	30.0	Timeout for the shared httpx client.
max_image_bytes	IMAGE_ANALYSIS_MAX_IMAGE_BYTES / VLM_MAX_IMAGE_BYTES	4194304 (4 MiB)	Max decoded image size; larger → 400.
max_new_tokens	IMAGE_ANALYSIS_MAX_NEW_TOKENS / VLM_MAX_NEW_TOKENS	256	max_tokens sent for /analyze.
warmup_image_path	WARMUP_IMAGE_PATH	./2fbe857fc85cf3d337365608e83f7a54.jpg	Warmup image; falls back to an embedded 1x1 PNG.
allowed_mime_types	IMAGE_ANALYSIS_ALLOWED_MIME_TYPES / VLM_ALLOWED_MIME_TYPES	image/png, image/jpeg, image/jpg, image/webp	MIME allowlist (comma-separated string accepted).

The serving port is not read by the app; it comes from AppConfig.image_analysis_port:

Setting	Env (aliases)	Default	Purpose
image_analysis_port	IMAGE_ANALYSIS_PORT / VLM_PORT	8012	Port the launcher serves the adapter on via uvicorn.

warning

device is reported only. This adapter does not enforce or probe CUDA — there is no CUDA-requirement setting on ImageAnalysisConfig. Likewise, VLM_GPU_UTIL is a launcher variable for the separate vLLM process, not read by this service.

Consumer

The service is called over HTTP by the conversation engine, which configures its own caller-side settings:

Setting	Default
image_analysis_url	http://localhost:8012/analyze
image_analysis_timeout_sec	15.0

note

The caller-side timeout (15.0s) is independent of the adapter's own backend timeout (vlm_timeout_sec, 30.0s).

Operational notes

• Concurrency — a single shared httpx.AsyncClient is reused for all requests; the service imposes no concurrency limit of its own. Any batching or limits live in the vLLM backend.
• CORS — wide open (allow_origins, allow_methods, allow_headers all ["*"]).
• Shutdown — the FastAPI lifespan calls service.aclose(), which closes the shared httpx client.