RAG Service

The RAG service is a stateless FastAPI microservice that performs retrieval-augmented retrieval only: it fetches and ranks grounding chunks from Qdrant but never generates an answer. Final prompt assembly and grounding decisions stay in the conversation engine.

Purpose & port

The service exposes a single POST /retrieve endpoint. Given a query and one or two Qdrant collection names, it rewrites the query, embeds it, searches the vector store, reranks the hits, and returns a ranked list of context chunks with citations and a low_confidence flag.

Property	Value
Module	component_services.rag_service.main:app
Port	8051 (set at launch from RAG_PORT, default 8051)
Consumed by	conversation_engine via RAGClient at ENGINE_CONFIG.rag_url (default http://localhost:8051/retrieve)
Persistence	None — reads only Qdrant collections, no relational tables
Auth	None on its own endpoints (a Bearer token is used only outbound to the rewrite LLM)

note

The RAG service is an internal HTTP dependency of the conversation engine. It is not proxied through the gateway, so its endpoints do not live under the /api_lis prefix.

Pipeline

POST /retrieve runs the following ordered pipeline:

1. Validate — reject an empty or whitespace-only query with RequestValidationError (HTTP 422).
2. Collection check — call collection_exists on collection_name and, when present, user_collection_name. If neither exists, raise CollectionNotFoundError (HTTP 404).
3. Query rewrite — rewrite the query with the LLM for retrieval quality (best-effort / fail-open).
4. Embed — embed the rewritten query via the TEI embeddings server.
5. Classify complexity — a static regex heuristic on the original query picks a token budget (simple / medium / complex).
6. Vector search — search every existing collection concurrently (asyncio.gather) using the request limit as top-k.
7. Merge & sort — flatten all chunks and sort by retrieval_score descending.
8. Retrieval floor — drop chunks below retrieval_score_floor (default 0.3).
9. Rerank — score the survivors with the TEI reranker and sort by rerank_score descending.
10. Confidence filter — drop chunks below confidence_threshold (default 0.5).
11. Assemble context — greedily pack chunks under a character budget (token_budget * 4), collect citations, and set low_confidence.

note

Query rewrite is fail-open: Processor.rewrite catches all exceptions and returns the original query, and an empty rewrite also falls back to the original. The remaining steps (embed, search, rerank) are fatal on failure — any unexpected error is wrapped as UpstreamError and returned as HTTP 503.

The complexity heuristic runs on the original query, independent of the rewritten query used for search:

Class	Condition	Budget (env)	Default tokens
Simple	≤ 10 words, no compare/multi markers, ≤ 1 ?	RAG_BUDGET_SIMPLE	512
Medium	≤ 28 words with compare/multi/question markers	RAG_BUDGET_MEDIUM	1024
Complex	everything else	RAG_BUDGET_COMPLEX	2048

Endpoints

Both routes are mounted directly on the app with no router prefix.

Method	Path	Description
GET	/health	Static liveness check. Returns {"service": "rag_service", "status": "ok"}. Does not probe dependencies.
POST	/retrieve	Runs the retrieve pipeline and returns a RAGContext.

POST /retrieve

Request body (RetrievalRequest):

{
  "query": "how do I reset my password",
  "collection_name": "avatar_42",
  "user_collection_name": "avatar_42_user_7",
  "limit": 15
}

Field	Type	Required	Default	Notes
query	string	yes	—	Rejected if empty/whitespace.
collection_name	string	yes	—	Base collection to search.
user_collection_name	string | null	no	null	Optional per-user collection.
limit	integer	no	15	Top-k per collection, 1–50.

Response body (RAGContext):

{
  "context_chunks": [
    {
      "chunk_id": "c-001",
      "text": "…retrieved passage…",
      "source_id": "doc-17",
      "collection_name": "avatar_42",
      "retrieval_score": 0.82,
      "rerank_score": 0.74
    }
  ],
  "citations": ["doc-17"],
  "low_confidence": false
}

Status	Cause
200	Pipeline completed (may return an empty context).
422	query is empty or whitespace.
404	None of the requested collections exist.
503	Upstream dependency or pipeline failure.

Scoring

Retrieval uses two independent score gates:

• retrieval_score_floor (default 0.3) — applied to the raw Qdrant score before reranking.
• confidence_threshold (default 0.5) — applied to the reranker score after reranking.

The score filter treats a missing score field as 0.0, so any chunk lacking the field is dropped by a positive threshold.

The low_confidence flag is True only when the final context is empty after the confidence filter — that is, when no chunk survived. When any chunk is returned, low_confidence is False.

Context assembly is budget-bounded: chunks with empty text are skipped, and chunks are packed greedily until the next chunk would exceed the character budget and at least one chunk is already kept. As a result the single highest-ranked chunk is always included, even if it alone exceeds the budget. citations is the sorted set of non-null source_id values across the kept chunks.

How the engine consumes the result

The conversation engine calls RAG concurrently with memory retrieval. On failure it emits a recoverable RAG_ERROR turn error and continues with an empty, low_confidence=True RAGContext, so a RAG outage does not break a turn. The engine only emits a RAGRetrievedEvent when the returned context is non-empty. For how the returned grounding drives the final prompt, see Conversation Pipeline.

Collections

Collection names are opaque to this service — it searches whatever names it is given and only searches the ones that exist. The per-avatar naming convention is built upstream by the engine's RAGClient:

Collection	Pattern	Notes
Base (avatar)	collection_name or avatar_{avatar_id}	Falls back to avatar_{avatar_id} when no explicit name is given.
Per-user	{base}_user_{user_id}	Included only when a user_id is present.

Each Qdrant point is expected to carry chunk_id, text, and source_id in its payload; the point's id and score supply the chunk id fallback and retrieval_score.

note

The engine's RAGClient posts only query, collection_name, and user_collection_name. It does not forward a limit, so the service always applies its default of 15.

Configuration

All settings come from the RAGConfig block. Most keys use a RAG_-prefixed alias with a shared fallback; the Qdrant keys use the shared QDRANT_* names directly.

Setting	Env (alias → fallback)	Default	Purpose
llm_endpoint	RAG_LLM_ENDPOINT → LLM_ENDPOINT	https://core-llm.nezlamna-online.education/api/generate	Full URL POSTed for query rewriting.
llm_api_token	RAG_LLM_API_TOKEN → LLM_API_TOKEN	change-me-in-production	Bearer token for the rewrite LLM.
llm_timeout_sec	RAG_LLM_TIMEOUT_SEC → LLM_TIMEOUT_SEC	30.0	Rewrite LLM request timeout.
rewrite_temperature	RAG_REWRITE_TEMPERATURE	0.0	Temperature for the rewrite LLM.
rewrite_max_tokens	RAG_REWRITE_MAX_TOKENS	128	max_tokens for the rewrite LLM.
embed_url	RAG_EMBED_URL → EMBED_URL	http://localhost:8016	TEI embeddings server base URL (POST /embed).
embedding_timeout_sec	RAG_EMBEDDING_TIMEOUT_SEC → EMBEDDING_TIMEOUT_SEC	10.0	Embedding request timeout.
qdrant_url	QDRANT_URL	http://localhost:6333	Qdrant vector store URL.
qdrant_api_key	QDRANT_API_KEY	"" (empty → None)	Optional Qdrant API key.
reranker_url	RAG_RERANKER_URL	http://localhost:8081	TEI reranker base URL (POST /rerank).
reranker_timeout_sec	RAG_RERANKER_TIMEOUT_SEC	10.0	Rerank request timeout.
retrieval_score_floor	RAG_RETRIEVAL_SCORE_FLOOR	0.3	Minimum Qdrant score before reranking.
confidence_threshold	RAG_CONFIDENCE_THRESHOLD	0.5	Minimum rerank score to keep a chunk.
budget_simple	RAG_BUDGET_SIMPLE	512	Token budget for simple queries.
budget_medium	RAG_BUDGET_MEDIUM	1024	Token budget for medium queries.
budget_complex	RAG_BUDGET_COMPLEX	2048	Token budget for complex queries.

Logging is configured at import time from LOG_LEVEL (default INFO) and LOG_FORMAT.

warning

embed_url defaults to http://localhost:8016, but the launch scripts start the TEI embeddings container on EMBED_PORT (default 8080) and do not export EMBED_URL / RAG_EMBED_URL. Unless a project .env overrides it, the RAG default embed port does not match the embeddings server port — set RAG_EMBED_URL (or EMBED_URL) explicitly for a working deployment.

Upstream dependency contracts

Dependency	Request	Response used
Rewrite LLM	POST to llm_endpoint with {prompt, system_prompt, temperature, max_tokens, stream:false}	json["text"] (markdown fences stripped)
Embeddings (TEI)	POST /embed with {"inputs": text}	json()[0] (first vector)
Reranker (TEI)	POST /rerank with {query, texts, truncate:true}	list of {index, score}
Qdrant	query_points on each collection	point payload + id + score

CORS is wide open (allow_origins, allow_methods, allow_headers all *). On shutdown, the FastAPI lifespan calls aclose(), which closes all four dependency clients.