In this guide
Mechanistic interpretability investigates how computations inside a neural network produce a particular behavior. Researchers identify possible components, describe their interactions and test parts of that explanation by changing internal activations. The useful result is a mechanism tied to evidence and a defined setting, with its unexplained parts still visible.
An answer can be correct while its explanation remains incomplete. A model's written chain of thought is generated text; investigating its internal computations requires different evidence. The readings below introduce that evidence through features, circuits and interventions.
Start without code. Follow the examples and selected sections first; leave equations and implementation appendices for a second pass. Three selections come from Anthropic, including a case study and its companion methods paper.
The parts of an explanation
An activation is a value computed inside a network. An interpretable feature is a component researchers associate with a recognizable pattern, such as a script or a place-related concept. Features need not line up with individual neurons: one neuron can respond to several unrelated patterns.
A circuit describes connected computations that contribute to behavior. Observing a feature activate suggests a hypothesis about its role. An intervention changes an internal value and tests the predicted effect. The claim depends on what was changed, what was measured and which assumptions the experiment held fixed.
1. See a circuit: Zoom In
Coauthored visual introduction and vision-model investigation · 10 March 2020
Zoom In: An Introduction to Circuits is by Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov and Shan Carter. Its curve detectors make an otherwise abstract question visible: which connected parts could explain a response?
Start with: the curve-detector example, its dataset and synthetic-image checks, the associated circuit, and the discussion of polysemantic neurons. The authors combine evidence rather than assigning a meaning from one striking image.
What you need: a network transforms its input through connected layers. No transformer knowledge is required here.
Keep the boundary: this is InceptionV1, a vision model. Use it to learn how a circuit hypothesis is investigated; its broader claims are explicitly speculative. The next reading supplies a separate language-model experiment.
2. Separate features from neurons: Towards Monosemanticity
Experimental language-model study · 4 October 2023
In Towards Monosemanticity: Decomposing Language Models With Dictionary Learning, Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn and their Anthropic coauthors, including Chris Olah, study a one-layer transformer.
A sparse autoencoder is another learned model that decomposes activations into sparsely used components. The paper investigates whether those components are easier to interpret than individual neurons, then tests their downstream effects.
Start with: the problem setup, the autoencoder explanation, the Arabic-script feature and its downstream effects. Finish with the discussion of how much behavior the interpretation explains.
What you need: distinguish a measured activation from the researcher's description of a feature. Equations can wait.
Keep the boundary: the example concerns a script used across languages, not a neuron that understands one language. This is a small language model; recovering its performance through a decomposition is not the same as understanding that fraction of its mechanism. Feature granularity also changes with dictionary size.
3. Test a component's role: Neel Nanda on patching
Author's technical explainer with worked experiments · 4 February 2023
Attribution Patching: Activation Patching At Industrial Scale explains a useful distinction between performing an intervention and approximating its effect.
Start with: Neel Nanda's activation-patching recap and John/Mary example. Two nearly identical prompts need different final names. Copying an activation from one run into the other lets the researcher measure how a particular answer score changes. Then read the attribution-patching approximation, experiments and limitations.
What you need: the two prompts define a contrast, and the chosen score defines what counts as an effect. You can follow this before the gradient formula.
Keep the boundary: attribution patching uses a fast gradient approximation to locate candidates for investigation. Its reliability varies by component, intervention and metric. It helps generate hypotheses; a promising score still needs a stronger test. Information passing through a component also does not establish that the component performed the whole computation.
4. Work through Dallas, Texas and Austin
Experimental case study of Claude 3.5 Haiku · 27 March 2025
On the Biology of a Large Language Model is a team paper led by Jack Lindsey; coauthors include Wes Gurnee, Emmanuel Ameisen and Chris Olah. Read its introductory example, inhibition experiments, feature swaps and limitations.
The prompt is “Fact: the capital of the state containing Dallas is”; Claude 3.5 Haiku answers Austin. Its interpretable replacement model's graph suggests Dallas-related features activate Texas-related features. These combine with capital-output features to favor Austin. A shortcut also links Dallas-related features toward saying Austin.
The authors test this hypothesis in the underlying model. Inhibiting Dallas-related features reduces Texas-related activity while largely preserving the capital-output group. They identify California-related features with an Oakland prompt. On the original Dallas prompt, suppressing Texas-related features and activating the California group changes the completion to Sacramento.
This supports an intermediate state representation's causal role in this example, alongside other paths. The graph belongs to a local replacement model; the interventions keep attention patterns fixed. Effects through changed attention remain untested.
What you need: a feature, a predicted connection and an intervention. The reported changed completion tests the proposed mechanism; it does not establish a complete explanation.
What each kind of evidence can show
Examples, graphs and interventions answer different questions about the same behavior.
- Activation examples Describe a feature
- Can show Examples help researchers propose what a feature responds to.
- Still open A useful label does not establish that the feature causes the answer.
- Attribution graph Propose a mechanism
- Can show Connections in a local replacement model suggest how features contribute to the selected output.
- Still open The approximation can miss computations or use mechanisms that differ from the underlying model.
- Feature intervention Test a prediction
- Can show Perturbing selected features tests predicted effects in the underlying model.
- Still open The result depends on the intervention setup, including the fixed attention patterns in this study.
Look for the claim being tested, the observed effect and the assumptions kept fixed.
5. Ask how the graph was built: Circuit Tracing
Methods and validation paper · 27 March 2025
Circuit Tracing: Revealing Computational Graphs in Language Models, by Emmanuel Ameisen, Jack Lindsey, Adam Pearce and their Anthropic coauthors, including Chris Olah, explains the machinery behind the case.
Start with: the introduction, replacement-model setup, validation with interventions, mechanistic faithfulness and limitations. A learned cross-layer transcoder approximates parts of the original computation using interpretable features. For a chosen input, a local version fixes attention patterns and adds corrections for the computation the learned components miss. Its graph concerns that prompt and target output token. Perturbation experiments check whether the proposed mechanisms carry over to the underlying model.
What you need: the Dallas example. Implementation appendices are optional on the first pass.
Keep the boundary: matching an output does not guarantee matching its mechanism. Reconstruction errors, omitted attention-pattern formation and choices about feature grouping leave gaps. An absent feature or missing graph connection does not establish that a computation never occurs. This tool does not certify a model's general safety.
What the Dallas result lets you conclude
You can now separate an interpretable label, a proposed connection and a tested effect. In the Dallas example, the authors went beyond naming a Texas-related feature: changing the selected state representation changed the completion in the predicted direction. The fixed-attention setup and other paths remain part of that conclusion.
For your next paper, keep a short record: which behavior is explained, which internal quantity was changed, what effect followed, and what the method leaves outside its scope. A diagram earns a bounded interpretation when the reported experiment supports the connection it claims.
Continue with Chris Olah's profile for the coauthored circuits and language-model work, or Neel Nanda's profile for patching explanations and research vocabulary. The AI safety collection connects interpretability with other safety questions; the LLM research collection offers a broader choice of authors and first materials. For experiments on the written steps themselves, see how reasoning models work.
How reasoning models work: five original reads
Twitter shadowban test: check search, replies and labels on X
From a vibe-coded prototype to a checked change