Contributors: Nicole Ren (GovTech), Ng Wei Cheng (GovTech)
chatbot that retrieves answers from your knowledge base. It’s reliable, grounded in factual content, and avoids the hallucination pitfalls that plague generative chatbots. Your system excels at fielding direct, standalone queries, but falters the moment users attempt natural, conversational follow-ups like “How can I do that?”, “What about in Germany?”.
The culprit? The typical retrieval mechanism treats each query in isolation, searching for semantic matches without any memory of the conversation’s context. While this approach works brilliantly for single-turn interactions, it breaks down when users expect the fluid, contextual dialogue that defines modern conversational AI.
What if you could give your bot memory? Welcome to the second part of our series exploring the GenAI innovations that power Virtual Intelligent Chat Assistant (VICA). VICA is a conversational AI platform designed to help government agencies respond to queries of citizens quickly whilst maintaining strict control over content through guardrails and custom-written responses that minimise hallucination. In our first article, we explored how LLM agents can enable conversational transactions in chatbots. Now, we turn our attention to another critical innovation: enabling natural, multi-turn conversations in stateless Q&A systems.
s
- The Anatomy of a Retrieval-Only Q&A Bot
- Conversational Breakdown: When Context is Lost
- The Solution: Query Rewriting with LLMs
- Conclusion
The Anatomy of a Retrieval-Only Q&A Bot
At its core, a retrieval-only Q&A chatbot operates on a beautifully simple premise: match user questions to pre-defined answers. This approach sidesteps the unpredictability of generative AI, delivering fast, accurate responses grounded in verified content. Here’s a breakdown of its components:
The Knowledge Base. Everything begins with a meticulously curated collection of question-and-answer pairs (Q&A) that forms the system’s knowledge base. Every question and its corresponding answer is written and vetted by a human expert. They are responsible for its accuracy, tone, and compliance. This “human-in-the-loop” approach means the bot is only ever reciting from a trusted, pre-approved script, eliminating the risk of it going off-message or providing incorrect information.
The Semantic Understanding Layer. The magic happens when these questions and answers are transformed into vector embeddings, which are dense numerical representations that capture semantic meaning in high-dimensional space. This transformation is crucial because it allows the system to understand questions like “How do I qualify for subsidies?” and “What is the criteria to apply for subsidies?” are essentially asking the same thing, despite using different words.
The Retrieval Engine. These embeddings are then indexed in a vector database (e.g. Pinecone) optimised for similarity search. When users ask questions, the system converts their query into a vector and performs a nearest-neighbour search, finding the most semantically similar match using metrics like cosine similarity or dot product, ranking potential matches and selecting the most relevant one.
The Response Delivery. Found the match? The system simply retrieves the corresponding pre-written answer. There’s no generation, no creativity, no risk of fabrication, just the delivery of verified, human-authored content.
The benefits are compelling: guaranteed accuracy, lightning-fast responses, and complete content control. But there’s a catch that becomes painfully obvious the moment users try to have an actual conversation.
Conversational Breakdown: When Context is Lost
Here’s where our retrieval system hits a wall. The moment users expect natural conversation, the stateless architecture reveals its Achilles’ heel: it treats every query as if it’s the first thing ever said.
The example above shows a realistic exchange that demonstrates this limitation perfectly. The bot handles the initial question flawlessly, retrieving accurate information about the weather today. But when the user asks a natural follow-up “What about in Germany?”, the system completely breaks down.
The follow-up question “What about in Germany?” lands nowhere near any meaningful match in the knowledge base without the context of “the weather today” from the previous exchange. This is the inevitable consequence of a stateless design.
The result? Users are forced to abandon natural conversational patterns and resort to repeating full context in every question: “How is the weather today in Germany?” instead of the intuitive “What about in Germany?”. This friction transforms what should feel like a helpful conversation into a frustrating game of twenty questions where you must start from scratch every time.
The Solution: Query Rewriting with LLMs
The solution is to use a separate, lightweight LLM to act as a query rewriter. This LLM intercepts user queries and, when necessary, rewrites ambiguous, conversational follow-ups into specific, standalone questions that the retrieval system can understand.
This introduces a preprocessing layer between user input and vector search. Here’s how it works:
Conversation Memory. The system now maintains a rolling window of recent conversation turns, typically the last 3–5 exchanges to balance context richness with processing efficiency.
Analyze and Rewrite where appropriate. When a new query arrives, use an LLM to examine both the current question and the conversation history to determine if rewriting is necessary. Standalone questions like “How is the weather today in Singapore?” pass through unchanged, whilst follow-up questions like “What about in Germany?” trigger the rewriting process. The rewritten context-rich query, “How is the weather today in Germany?” is then passed to the vector database as per usual.
Fallback (Optional). If the rewritten query fails to find good matches, the system can fall back to searching with the original user query, ensuring robustness.
Conclusion
The transformation from amnesia to awareness doesn’t require rebuilding your entire chatbot architecture. By introducing query rewriting with LLMs, we’ve given retrieval-only chatbots the one thing they were missing: conversational memory. This hybrid approach of inserting an LLM as an intelligent query rewriter gives your chatbot the memory it needs to handle natural, multi-turn conversations while preserving the content control that made retrieval-only chatbots attractive. The result is a chatbot that feels genuinely conversational to users but remains grounded in verified, human-authored responses. Sometimes the most powerful solutions are not revolutionary overhauls, but thoughtful enhancements that address a critical missing piece, thereby transforming the whole.
Curious about VICA? Head over to read our first article on enabling conversational transactions in chatbots if you haven’t! If you are a Singapore public service officer, you can visit our website at https://www.vica.gov.sg to create your own chatbot and find out more!