Lil'Log Notify me• Jul 7, 2024essay

Extrinsic Hallucinations in LLMs

Lilian Weng(guest)

Get the full transcript next time Lil'Log releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on Lilian Weng.

The Paradox of Fine-Tuning The Evolution of Hallucination Detection

12 quotes

Concerns Raised

Fine-tuning on new knowledge can paradoxically increase hallucination rates.
Standard RLHF alignment can degrade model calibration and factuality.
Larger models are not inherently more truthful and can amplify common misconceptions.
Reinforcement learning has shown limited or no improvement for factuality in some key experiments (e.g., WebGPT, GopherCite).

Opportunities Identified

Advanced detection frameworks like SAFE can automate factuality evaluation at a scale and accuracy rivaling human annotators.
Inference-time strategies like Chain-of-Verification (CoVe) and Self-RAG can reduce hallucinations without costly retraining.
Factuality-specific fine-tuning methods, such as DPO on factuality-based preferences, offer a direct path to improving truthfulness.
Using LLMs as agents for verification and research is a promising and cost-effective approach to ensuring accuracy.

Key Themes

Research Findings12

Once LLMs learn examples with new knowledge through fine-tuning, their tendency to hallucinate increases, according to Gekhman et al. (2024).

The SAFE factuality evaluation method achieved a 72% agreement rate with human annotators and a 76% win rate over humans when they disagreed.

On the TruthfulQA benchmark, larger models were found to be less truthful due to their tendency to learn common misconceptions.

RLHF fine-tuning makes LLMs poorly calibrated, but using a higher sampling temperature can lead to better calibration results.

RLHF can worsen factuality because human feedback often prefers longer, more detailed answers, which are not necessarily more factual.

In-context hallucination in large language models occurs when the model's output is inconsistent with the provided source content.

Extrinsic hallucination in large language models occurs when the model's output is fabricated and not grounded by its pre-training data or world knowledge.

According to a study by Gekhman et al. (2024), LLMs learn fine-tuning examples with new knowledge slower than examples consistent with their pre-existing knowledge.

The FActScore method decomposes a long-form generation into multiple atomic facts and validates each one against a knowledge base like Wikipedia.

In biography generation tasks, LLM error rates are higher for rarer entities and for facts mentioned later in the generation.

The SAFE (Search-Augmented Factuality Evaluator) method uses a language model as an agent to iteratively issue Google Search queries to verify atomic facts.

SelfCheckGPT detects factuality mistakes by checking for consistency across multiple samples from a black-box LLM, without requiring an external knowledge base.

Topics

LLM Hallucination Extrinsic Hallucination Factuality Model Calibration Retrieval-Augmented Generation (RAG)Fine-tuning Reinforcement Learning from Human Feedback (RLHF)Hallucination Detection Self-RAG Chain-of-Verification (CoVe)SAFE (Search-Augmented Factuality Evaluator)FActScore TruthfulQA AI Safety Model Evaluation

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

Extrinsic Hallucinations in LLMs

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

The Paradox of Fine-Tuning

The Evolution of Hallucination Detection

Advanced Mitigation Beyond RAG

The Calibration and Truthfulness Dilemma

Research Findings12

Topics