Asking the LLM to self-diagnose and self-correct the prompt to improve answer quality.
Retrieval Augmented Generation (RAG) is undoubtedly a powerful tool, easily crafted using frameworks like LangChain or LlamaIndex. Such ease of integration might give an impression that RAG is a magic solution that is easy to build for every use case. However, in our journey to upgrade our editorial article search tool to offer semantically richer search results and direct answers to queries, we found the basic RAG setup and is lacking and discovered many challenges. Constructing a RAG for a demonstration is quick and easy, often yielding sufficiently impressive results for small subset of scenarios. Yet, the final stretch to achieve production-ready status, where exceptional quality is mandatory, presents significant challenges. This is particularly true when dealing with a vast knowledge base filled with thousands of domain-specific articles, a not-so-rare occurrence.
Our approach to RAG consists of two distinct steps:
- Relevant Document Retrieval By employing a fuse of dense and sparse embeddings, we extract relevant document chunks from our Pinecone database, considering both content and title. These chunks are subsequently re-ranked based on relevance to the title, content, and the document’s age. The top four documents are then chosen: both as potential search results and as document context for generating direct answers. Notably, this approach diverges from the common RAG setup, addressing our unique document retrieval challenges more effectively.
- Direct Answer Generation Here, the question, instruction, and the previously retrieved top four document chunks (document context) are fed into a Large Language Model (LLM) to produce a direct answer.
I’ve delved deeply into enhancing document retrieval quality through the use of Hybrid Search and Hierarchical Document Ranking techniques in previous discussions. In this blog, I aim to share insights on refining and troubleshooting the…