Production-grade RAG system over FastAPI's official documentation
Caught a silent data-coverage bug in ChromaDB ingestion that was leaving collections sparsely populated
Context
Most portfolio RAG projects are a thin wrapper: load a PDF, embed it, ask an LLM a question, and call it done. That doesn't demonstrate real-world engineering competence. Retryv was built to go one layer deeper—hitting and resolving the silent failure modes that occur when you try to make retrieval reliable, such as sparse ingestion, hallucination control, and embedding quota limits.
The Problem
FastAPI's documentation is large and deeply nested, making it easy for developers to get lost. Retryv indexes the full documentation and answers natural-language questions grounded in actual doc content, with a built-in confidence mechanism that prevents the model from generating plausible-sounding but ungrounded answers.
Technical Approach
FastAPI docs are chunked and embedded using the Gemini API and stored in ChromaDB. Retrieval uses a hybrid mechanism (BM25 sparse keyword search + dense vector search combined using Reciprocal Rank Fusion) to ensure both exact keyword and semantic matches are returned. An explicit RRF score threshold (0.025) acts as a confidence guard to prevent hallucinated answers when retrieval confidence is too low.
Key Engineering Decisions
Dense vector retrieval (embeddings) often misses exact-keyword queries (like specific parameters or error codes), while BM25 misses semantic questions. Blending both via Reciprocal Rank Fusion (RRF) avoids picking one failure mode over the other.
Most RAG demos always generate an answer even when retrieval is weak, producing hallucinations. Gating answers behind a tuned RRF confidence threshold (0.025) ensures the system says 'I don't know' rather than hallucinating.
- Silent Ingestion Bug: Ingestion failures left ChromaDB collections sparsely populated without throwing an exception. Invalidated chunking-strategy comparisons until identified and fixed.
- Silent Zip-Truncation in Gemini Embedding: A batching bug in the Gemini embedding call was silently dropping data on large runs, caught by comparing raw input counts against finished vector records.
- Missing BM25 Index Files: The hybrid retrieval was silently falling back to vector-only search because BM25 index files weren't being generated correctly in some runs, caught by seeing identical dense and hybrid outputs.