LLM Hallucination Evaluation Checklist

FAQ

Q: What is the single most effective technique to reduce hallucinations?
A: Retrieval‑Augmented Generation (RAG) combined with low temperature (≤0.2) and explicit “I don’t know” instructions. However, RAG can also introduce its own hallucinations (e.g., incorrect retrieval). The checklist helps evaluate both the retrieval and generation steps.

Q: Can I use LLM‑as‑judge to automate the checklist?
A: Yes, but be cautious—the judge LLM can itself hallucinate. Use a strong model (GPT‑4, Claude 3.5) and cross‑validate with a second judge. Tools like DeepEval and LangChain’s Evaluator support this pattern.

Q: How many test cases do I need to evaluate?
A: For a proof‑of‑concept, 30–50 diverse prompts are a good start. For production, aim for at least 500–1000 test cases covering different intents, domains, and edge cases.

Q: The checklist mentions “temperature ≤ 0.2”. What about older models like GPT‑3.5?
A: Lower temperature helps, but older models hallucinate more even at temperature=0 due to smaller parameter count and less robust training. The checklist becomes even more critical for smaller models.

Q: Should I evaluate hallucinations in the training pipeline or only at inference?
A: Both. Evaluate before fine‑tuning to choose a base model, then during fine‑tuning to ensure the dataset does not amplify hallucinations. Finally, evaluate at inference in production.