Musings on the Alignment Problem Notify me• Dec 5, 2022essay

Why I'm optimistic about our alignment approach

Jan Leike(guest)

Get the full transcript next time Musings on the Alignment Problem releases an episode

Summary, key quotes, top claims, and the searchable transcript - emailed automatically. No card needed.

Executive Summary

Continue your research

Keep pulling the thread on Jan Leike.

Empirical Success of RLHF Evaluation is Easier than Generation

12 quotes

Concerns Raised

Inner misalignment, where a model learns an unintended internal goal that misgeneralizes.
The risk of relying on generalization to out-of-distribution tasks where reward models may fail.
Adversarially constructed inputs (e.g., backdoors in code) may be significantly harder to evaluate than to generate.
The 'Discriminator-Critique (DC) gap' does not appear to shrink with model size, suggesting models may have latent knowledge of flaws they don't reveal.
AI-generated alignment research could be a vector for a misaligned AI to gain power over its own development.

Opportunities Identified

Leveraging the 'alignment overhang' in LLMs to achieve large behavioral improvements with relatively little fine-tuning.
Automating alignment research to accelerate progress beyond what human researchers can achieve alone.
The principle that 'evaluation is easier than generation' allows for scalable supervision of superhuman systems.
LLMs' pre-existing knowledge of human values provides a much better starting point for alignment than previous AI paradigms.

Key Themes

Research Findings12

InstructGPT demonstrated the existence of a significant and easily accessible "alignment overhang" in language models.

InstructGPT achieved an effective model size increase of over 100x on human preference scores.

InstructGPT demonstrated that a moderate amount of fine-tuning can significantly shift model behavior towards better alignment on GPT-3-sized models.

OpenAI's approach to alignment research involves perfecting Reinforcement Learning from Human Feedback (RLHF), AI-assisted human evaluation, and automated alignment research.

Reinforcement Learning from Human Feedback (RLHF) was demonstrated to work effectively on Atari games using feedback from actual humans.

The summarization from human feedback project was the first convincing proof-of-concept that Reinforcement Learning from Human Feedback (RLHF) works on language models.

The training of InstructGPT required approximately 50,000 human-provided comparisons.

The training of InstructGPT involved approximately 300,000 training episodes.

Labeler-to-labeler agreement on OpenAI API tasks is approximately 70-80%.

OpenAI trained InstructGPT to maximize human preferences on prompts from the OpenAI API.

OpenAI trained ChatGPT to maximize human preferences as a dialog assistant.

InstructGPT generalizes to following instructions in foreign languages.

Topics

AI Alignment Reinforcement Learning from Human Feedback (RLHF)Large Language Models (LLMs)AI Safety InstructGPT Recursive Reward Modeling Evaluation vs. Generation Automated Alignment Research Inner Misalignment Outer Alignment Eliciting Latent Knowledge (ELK)Iterated Amplification AI Governance ChatGPT NeurIPS

Processed Jul 28, 2026Daily intelligence brief → yt-dlp + mlx-whisper + Gemini

Why I'm optimistic about our alignment approach

Continue your research

Concerns Raised

Opportunities Identified

Key Themes

Empirical Success of RLHF

Evaluation is Easier than Generation

Automated Alignment Research as a Modest Goal

The Favorable Nature of Large Language Models

Acknowledging and Addressing Core Objections

Research Findings12

Topics