Keep pulling the thread on Chris Olah.
The success of mechanistic interpretability for AI safety is dependent on solving the problem of superposition.
Induction heads are a type of attention head in transformers that enable a form of in-context learning by searching for previous patterns and copying the subsequent token.
The formation of induction heads during training creates a visible bump in the loss curve of every transformer model with more than two layers that the speaker has observed.
The formation of induction heads in transformer models causes a deviation in their scaling laws.
The two major challenges for using mechanistic interpretability to provide safety guarantees are superposition and scalability.
The goal of mechanistic interpretability is to reverse engineer neural network parameters into a human-readable format analogous to source code.
Reverse engineering a complex neural network is likely a harder challenge than reverse engineering a program like the Linux kernel.
In vision models, a 'car detector' circuit can be constructed from lower-level feature detectors for windows and wheels, with weights that encode the spatial relationship of windows being above wheels.
Superposition occurs when a neural network represents more features than it has neurons, preventing a one-to-one mapping between features and individual neurons.
A neural network with superposition can be conceptualized as a simulation of a larger, sparser neural network that has been projected down and folded onto the smaller, observed network.
In toy models, as features become sparser, a neural network can represent more features than it has dimensions, with experiments showing the ability to represent 5 features in a 2-dimensional space.
In toy models, features learned in superposition organize themselves into geometric structures corresponding to regular or uniform polyhedra.