Project — Flagship 1: a RAG + agent service with evals and tracing

Every exercise on this phase built one piece in isolation — BM25 ranking, RRF fusion, a citation checker, an MCP tool server, a RAGAS-style harness against a mock judge. This project wires them into one real service: ingest a document set into Postgres with pgvector, retrieve hybrid (BM25 + vectors, fused and re-ranked), answer through an agent loop that calls at least two tools, trace every call in Langfuse, and grade the whole thing with a RAGAS report and a citation check over 50 real answers. It runs on your machine and, later, on Fargate — this page is the checklist, not a tutorial to copy.

~12 hruns locally · python + docker 8 milestonesf1-build

What you're building

Ingest — chunk a real document set (your own corpus, not a toy fixture) and write chunks + embeddings into Postgres with the pgvector extension.
Hybrid retrieval — BM25 over the chunks (the ranker from exercise-bm25) fused with vector search via RRF, then diversified with MMR (both from exercise-rrf-mmr) — wire your own ranking functions in, not a vector-store's built-in hybrid mode.
Agent loop — a loop that plans, calls tools, and answers, exposing at least two tools over MCP with schema validation (the server from exercise-mcp-server).
Tracing — every retrieval and every tool call visible as a Langfuse trace, not just logged to stdout.
Evals — a RAGAS-style report (exercise-ragas-mock's faithfulness / answer-relevance / context-precision; a real LLM judge is optional this time) and a citation-faithfulness check (exercise-citation-check) run over 50 real answers from the service, not the exercises' fixtures.

The milestones

Repo public — the service lives in its own public repo, linked from your profile.
Ingest + pgvector — a real corpus chunked and embedded into a Postgres/pgvector table you can query.
Hybrid retrieval — BM25 and vector rankings fused with RRF and re-ranked with MMR, both reusing the functions you already built and tested.
Agent loop with two tools — the loop calls at least two MCP tools (reusing the schema-validated server) as part of answering, not as a demo bolted on afterward.
Langfuse traces visible — open Langfuse and see the retrieval + tool calls for a real request, not a screenshot of a single mocked one.
RAGAS report — faithfulness, answer relevance and context precision computed over real service answers (the mock judge from exercise-ragas-mock is fine; a real judge is optional and costs money).
Citation check on 50 answers — the exercise-citation-check logic run over 50 real answers from the service, with the supported / unsupported / wrong-number counts reported.
README with the eval table — the RAGAS and citation numbers in the repo's README, next to what corpus and how many chunks they were measured on.

Suggested build order

Ingest first, and check it by hand — query a few chunks back before wiring anything else to it.
Hybrid retrieval next, against the ingested corpus — this is where exercise-bm25 and exercise-rrf-mmr move from a browser exercise into service code.
The agent loop and its two MCP tools, once retrieval returns real chunks to reason over.
Langfuse tracing wrapped around the retrieval and tool calls you now have, so every later change is visible in a trace, not just believed.
The evals last, against 50 real answers the running service produced — a RAGAS report and a citation check computed on fixtures would not prove anything about the service.
Containerizing (docker compose: api + pgvector + Langfuse) and a Fargate deployment follow as their own cloud guides later on this track; this milestone list is satisfied by the service running locally first.

All 8 milestones above are done and committed to the public repo.

This is self-attestation — the site cannot see your repo or your Langfuse project, so the box and the button are you telling The Path the service exists and the milestones are real.

If you get stuck

pgvector — the official Docker image (pgvector/pgvector) is the fastest local start; CREATE EXTENSION vector; once per database, then an embedding vector(N) column sized to your embedding model's output.
Langfuse — self-host with its own docker compose file, or use the hosted free tier for a solo project; either way, wrap retrieval and tool calls with its SDK's decorators rather than adding tracing after the fact.
Real judge cost — a real LLM judge for RAGAS is optional for this milestone; the mock judge from exercise-ragas-mock keeps the report exact and free, and the README can say so.
50 answers — draw them from real questions against your ingested corpus, not 50 variations of the same question — the citation check and RAGAS numbers mean more spread across the corpus than repeated on one slice of it.