5 Techniques to Prevent Hallucinations in Your RAG Question Answering

problem when working with LLMs. They are a problem for two main reasons. The first apparent reason is that a hallucination naturally causes the user to receive an incorrect response. The second, arguably worse reason, is that hallucinations lower the users’ trust in the system. Without the user believing in your question answering system, it will be difficult to keep the users on your platform.

Table of contents

Why do you need to minimize hallucinations?

When clients think about using an LLM to solve a problem, one of the first thoughts that often comes to mind is hallucinations. They’ve often heard that LLMs sometimes output text that isn’t true, or at least answers you cannot fully trust.

Unfortunately, they are often correct, and you need to take steps to minimize hallucinations in your question answering systems. The contents of this article will refer to hallucinations specifically for question answering systems, though all of the techniques can be applied to other LLM applications as well, such as:

• Classification
• Information extraction
• Automations

Throughout this article, I’ll discuss real-world techniques you can apply to mitigate the impact of hallucinations, either by preventing them from happening at all or by minimizing the damage a hallucination causes. This can, for example, be damage such as the user trusting your application less after experiencing

Techniques to prevent hallucinations

I’ll separate this section into two subsections:

1. Techniques to directly lower the amount of hallucinations you experience from the LLMs
2. Techniques to mitigate the damage of hallucinations

I do this because I think it’s helpful to get an overview of techniques you can utilize to minimize the impact of hallucinations. You can either try to prevent them from happening at all (the first technique), though this is near impossible to achieve with 100% accuracy, or you can mitigate the damage of a hallucination once it happens (technique number two).

Lower the amount of hallucinations

In this subsection, I’ll cover the following techniques:

• Verification step (LLM judge)
• RAG improvements
• Optimize your system prompt

LLM judge verification

The first technique I’ll cover is utilizing LLM as a judge to verify your LLM responses. This technique relies on the concept below:

Verifying a response, is usually a simpler task than generating the response

This quote is sometimes more easily understood for math problems, where coming up with a solution is often rather difficult, but verifying that the solution is correct is a lot simpler. However, this concept also applies to your question answering system.

To generate a response, an LLM has to read through a lot of text and interpret the user’s question. The LLM then has to come up with a suitable response, given the context it’s been fed. Verifying the answer, however, is usually easier, considering the LLM verifier needs to judge whether the final response makes sense given the question and the context. You can read more about LLM validation in my article on Large Scale LLM Output Validation.

RAG improvements

There are also a lot of improvements you can make to your RAG pipeline to prevent hallucinations. Step number one is to fetch the correct documents. I recently wrote about this process, with techniques to both increase the precision and recall of the documents you fetch for RAG. It mainly boils down to filtering away potentially irrelevant documents (increasing precision) through techniques like reranking and LLM verification. On the other side, you can ensure to include relevant documents (increase recall), with techniques such as contextual retrieval and fetching more document chunks.

Optimize your system prompt

Another technique you can utilize to lower the amount of hallucinations is to improve your system prompt. Anthropic recently wrote an article about writing tools for AI agents, highlighting how they use Claude Code to optimize their prompts. I recommend doing something similar, where you feed all your prompts through an LLM, asking the LLM to improve your prompt in general, and also highlighting cases where your prompt succeeds and where it fails.

Furthermore, you should include a sentence in your prompt highlighting that the LLM should only utilize the provided information to answer the user’s question. You want to prevent the model from coming up with information based on its pre-training, and rather utilize the context it’s been provided.

prompt = f"""


You should only respond to the user question, given information provided 
in the documents below.

Documents: {documents}
User question: {question}
"""

This will drastically reduce how often the LLM responds based on its pre-training, which is a common source of hallucinations for LLMs.

Mitigate the damage of hallucinations

In the previous subsection, I covered techniques to prevent hallucinations from happening. However, sometimes the LLM will still hallucinate, and you need measures in place to mitigate the damage when this happens. In this subsection, I’ll cover the following techniques, which help minimize the impact of hallucinations:

• Citing your sources
• Help the user utilize the system effectively

Citing your sources

One powerful technique is to make the LLM cite its sources when providing answers. You can, for example, see this whenever you ask ChatGPT to answer a question, based on content on the internet. ChatGPT will provide you with the response, and after the response text, you can also see a citation pointing to the website from which the answer was taken.

You can do the same for your RAG system, either live while answering the question or in post-processing. To do it live, you can, for example, give individual IDs to each document chunk, and ask the LLM to cite which document chunk it used to cite the answer. If you want higher quality citations, you can do it in post-processing, which you can read more about in this Anthropic Docs.

Guiding the user

I also think it’s worth mentioning how you should guide the user to efficiently utilize your application. You, as the creator of your RAG question answering system, know exactly its strengths and weaknesses. Maybe your system is amazing at answering one type of question, but performs worse on other question types.

When this is the case (which it very often is), I highly recommend you inform the user of this. The counter argument here is that you don’t want the user to know about your system’s weaknesses. However, I would argue it’s way better to inform the user of these weaknesses beforehand, rather than the user experiencing the weakness themselves, for example, through a hallucination.

Thus, you should either have an info text around your question answering system, or an onboarding flow, which informs the user that:

1. The model works very well, but can occasionally make mistakes
2. What kind of questions is the model good at answering, and what types of questions are you working on improving

Summary

In this article, I have discussed hallucinations for LLMs. Hallucinations are a massive problem, which a lot of users are aware of. You should thus have several specific measures in place to mitigate hallucinations. I have covered techniques to outright reduce hallucinations, such as improving RAG document fetching, optimizing your system prompt, and LLM validation. Furthermore, I also discussed how you can minimize the damage of hallucinations once they happen, for example, by providing citations to sources and guiding the user to use your application effectively.

👉 Find me on socials:

🧑‍💻 Get in touch

🔗 LinkedIn

🐦 X / Twitter

✍️ Medium