Finding the Right Answers from Thousands of Documents: A Smarter RAG Approach

Author(s): Shrinidhi Atmakur Originally published on Towards AI. Finding the Right Answers from Thousands of Documents: A Smarter RAG Approach Introduction RAG is often presented as a simple, three-step architecture: put documents into a vector database, convert the user’s question into an embedding, retrieve a handful of chunks, and hand them to an LLM. That approach is a great proof of concept. It is also where most RAG projects quietly stall. But what happens when the knowledge base grows to hundreds or thousands of documents? Retrieval becomes more challenging, irrelevant chunks can reach the LLM, token consumption increases, and the quality of the final answer becomes increasingly dependent on retrieval quality. This article shares information on building a production-ready, multi-stage RAG pipeline that can scale without simply sending more and more context to the LLM. RAG in Two Lines Retrieval-Augmented Generation allows an AI model to answer questions using external knowledge. Instead of relying only on the LLM’s internal knowledge, the system retrieves relevant information from a knowledge base and provides it to the model as context before generating an answer. User Question → Retrieve Relevant Context → LLM → Answer The important word here is relevant.A powerful LLM cannot consistently produce high-quality answers if the retrieval system provides incomplete or irrelevant context. The Problem With Simple Vector Search A typical RAG implementation looks like this Documents → Chunk Documents → Create Embeddings → Store in Vector DB And on the query side:User Question → Create Query Embedding → Vector Search → Retrieve Chunks → Send to LLM → Generate Answer For a small dataset, this approach may work perfectly well. However, as the knowledge base grows, several challenges start to appear. ScalingWith thousands of documents, there may be hundreds of thousands of chunks. A vector search may return content that is semantically similar but does not actually answer the user’s question. Response TimeOne common solution is to retrieve more chunks. But more chunks mean more processing and potentially more context sent to the LLM. Token ConsumptionNot every retrieved chunk is useful. Passing 30 or 50 chunks directly to the LLM can significantly increase token consumption while adding unnecessary noise. Response QualitySemantic similarity does not always mean answer relevance. The 4-Stage Hybrid Retrieval Pipeline The overall philosophy is:Retrieve broadly. Combine intelligently. Rerank precisely. Then let the LLM reason. Stage 1: Embedding and Candidate Retrieval The first stage focuses on recall. The objective is not necessarily to find the perfect chunks immediately. Instead, the objective is to identify a wider set of potentially relevant candidates. Documents are split into chunks and converted into embeddings. These embeddings are stored in a vector database such as ChromaDB. When a user submits a question, the query is converted into an embedding using a model such as: all-MiniLM-L6-v2 This model produces vector representation of the text. The vector database then searches for semantically similar chunks.For example:User Query: “How do I rotate secrets?”Query Embedding -> Vector Database ->Top 50 Candidate Chunks The important idea is to retrieve a wider net of candidates. Vector search is fast and excellent at identifying semantic similarity, making it an ideal first stage. But semantic search should not be the only retrieval mechanism. Stage 2: BM25 and Reciprocal Rank Fusion Vector search is good at understanding semantic meaning, but it may miss exact keywords, error codes, commands, or technical terms. BM25 complements vector search by performing keyword-based retrieval. Instead of choosing one approach, the results from both searches are combined using Reciprocal Rank Fusion (RRF). RRF uses the ranking position of each result rather than directly comparing scores from different retrieval methods.Vector Search + BM25 → RRF Fusion → Better Candidate Ranking This creates a hybrid retrieval mechanism that combines semantic understanding with exact keyword matching. Stage 3: Cross-Encoder Reranking After the first two stages, the pipeline may have reduced thousands of chunks to perhaps 20 or 30 strong candidates. The next question is: Which of these chunks actually answers the user’s question? This is where a cross-encoder becomes useful. An embedding model processes the query and document separately and compares their vector representations. A cross-encoder processes them together:Query + Candidate Chunk → Cross-Encoder → Relevance Score For example:Query: “How do I rotate Cloud secrets?” Candidate Scores:0.98 Cloud Secret Manager supports automatic rotation…0.81 Secret lifecycle defines credential management…0.22 Cloud provides several storage services…0.07 Metadata helps organize enterprise data… The cross-encoder can make a much more precise relevance decision because it sees the query and candidate chunk together. The trade-off is performance. A cross-encoder is slower than vector search, so running it against thousands of chunks would be inefficient. That is why the earlier stages are important:100,000 Chunks → Vector + BM25 Retrieval → 30 Candidates → Cross-Encoder → Top 5 This follows a simple principle: Use fast retrieval to reduce the search space, then use more precise models on a smaller candidate set. Stage 4: LLM Answer Generation Only after the retrieval and ranking stages do we send context to the LLM. The final top-ranked chunks are assembled into a context prompt and passed to Mistral, or any other LLM.Top Relevant Chunks → Context Builder → Mistral / LLM → Final Answer A simple prompt could instruct the model to:* Answer using only the provided context.* Avoid making unsupported claims.* Clearly state when the answer cannot be found.* Provide source information where possible. The LLM can now focus on what it does best: reasoning, connecting information, summarizing, and generating a clear response. It does not need to search through 50 loosely related chunks. How the Complete Pipeline Works Imagine a knowledge base containing:2,000 Documents -> 100,000 ChunksA user asks: How can I troubleshoot a failed secret rotation?The pipeline could work like this: Every stage has a specific purpose. Instead of asking one component to do everything, the pipeline allows each component to do what it is best at. Best Practices for Fine-Tuning the Pipeline A RAG pipeline should be tuned based on the type and size of the knowledge base. […]