Berkeley AI series Notify me• Apr 20, 2023• 1:03:15Interview

John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges

From Berkeley AI series

John Schulman(Co-founder, OpenAI; Chief Architect, ChatGPT, guest)

Get the full transcript next time Berkeley AI series releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on John Schulman.

The Hallucination Problem in LLMs Limitations of Supervised Fine-Tuning for Truthfulness

11 quotes

Concerns Raised

Supervised fine-tuning (behavior cloning) can inadvertently train models to hallucinate rather than be truthful.
It is becoming prohibitively difficult for human labelers to check long, complex model outputs for factual errors, creating a need for scalable oversight.
RLHF may optimize for what sounds convincing to a human rather than what is factually correct.
There is a strong precision-recall conflict between training a model to be informative and training it to be correct.

Opportunities Identified

Using reinforcement learning to effectively teach models the boundary of their knowledge and when to express uncertainty.
Implementing retrieval and source citation to make model outputs verifiable, which is crucial for training and user trust.
Developing model-assisted evaluation techniques to create scalable oversight for complex tasks.
Using a model's 'inner monologue' to improve interpretability and enable more effective, shorter-horizon RL training.

Key Themes

Research Findings12

Proximal Policy Optimization (PPO) is the most widely used algorithm in its space and is used as part of ChatGPT's training.

Truthfulness is one of the biggest technical problems for language models today.

Training a language model with behavior cloning (supervised fine-tuning) on 100% correct answers teaches the model to hallucinate when it lacks the underlying knowledge for a given question.

Training a model to say "I don't know" based on human labelers who don't know the answer can inadvertently train the model to withhold information that it actually possesses.

It is impossible to train a model to be truthful with behavior cloning unless you can look inside the model to see what it knows.

Fine-tuning open source base language models on ChatGPT outputs will likely result in models with significant factual accuracy problems.

The most important reason for using retrieval and citing sources in language models is verifiability for human checkers.

The ChatGPT browsing model only browses when it determines it does not know the answer, using its internal sense of uncertainty.

A major open problem in AI is how to incentivize a model to accurately express its uncertainty in words, including the right amount of hedging.

It is becoming necessary to develop scalable oversight techniques because it is getting very difficult for human labelers to keep up with model capabilities.

A speculative problem with RLHF is that it optimizes for human approval and what sounds convincing, not necessarily for objective truth.

John Schulman is a co-founder of OpenAI and the chief architect of ChatGPT.

Topics

Reinforcement Learning (RL)Reinforcement Learning from Human Feedback (RLHF)Language Models (LLMs)Truthfulness Hallucination Factuality Supervised Fine-Tuning (SFT)Behavior Cloning Proximal Policy Optimization (PPO)Trust Region Policy Optimization (TRPO)Retrieval-Augmented Generation (RAG)Scalable Oversight Model Uncertainty ChatGPT GPT-4 WebGPT AI Safety Interpretability Reward Modeling

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

John Schulman - Reinforcement Learning from Human Feedback: Progress and Challenges

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

The Hallucination Problem in LLMs

Limitations of Supervised Fine-Tuning for Truthfulness

Reinforcement Learning as a Solution for Factuality

Retrieval and Verifiability

Scalable Oversight and Open Problems

Research Findings12

Topics