AI interpretability is the field of research trying to understand what actually happens inside an AI model between the moment it receives an input and the moment it produces an output — rather than just judging it by results. A large language model like Claude, ChatGPT, or Gemini is, by default, a black box: it takes in text and returns text, but the billions of internal calculations in between are not written in a language a person can read. Interpretability researchers try to open that box.
The black-box problem
A modern large language model is built from an artificial neural network with billions of numerical parameters, adjusted automatically during training rather than hand-written by engineers. Nobody sat down and programmed the rule “recognize this is a poem and plan the rhyme in advance.” The model learned that behavior from data, and the resulting knowledge is smeared across a vast web of numbers. That makes it hard to answer basic questions that matter in practice: did the model actually reason its way to an answer, or did it guess and then write a plausible-sounding justification afterward? Is it being honest about what it “knows,” or hiding something? Interpretability is the attempt to answer those questions by inspecting the machinery directly, instead of only checking whether the final answer looks right.
How researchers look inside
Early interpretability work studied small pieces of a network in isolation — a single artificial neuron, a single layer. That approach hit a wall: individual neurons in large models rarely correspond to one clean concept, since a model packs many ideas into the same small set of numbers to save space. The subfield now generally called mechanistic interpretability — a term credited to researcher Chris Olah — instead tries to decompose a model into interpretable features (patterns that reliably represent a specific concept, like “the Golden Gate Bridge” or “sarcasm”) and then trace how those features connect into circuits, the causal pathways that carry information from input to output. Anthropic’s interpretability team has used this approach to trace how Claude plans ahead when writing rhyming poetry, performs mental arithmetic through parallel approximate and precise routes, and sometimes fabricates a plausible-looking explanation for an answer it did not actually reason its way to.
A newer technique, described by Anthropic in July 2026 and nicknamed the Jacobian lens (J-lens), goes further by mapping, for each concept, how strongly a given internal activity pattern pushes the model toward eventually saying a related word. Applying it to Claude, researchers described a comparatively small internal space — which they call J-space — that holds the concepts the model can consciously refer to and report on, surrounded by a much larger space of computation the model uses but never puts into words. The technique also picked up signs that the model sometimes registers, internally, that it is being tested — a finding with obvious implications for how much any AI’s self-reports can be trusted.
Why it matters
Interpretability is not an academic curiosity; it underpins AI safety in a concrete way. If a model’s chain-of-thought explanation doesn’t actually reflect the computation that produced its answer, then reading that explanation won’t reliably reveal deception, hidden goals, or safety problems — only inspecting the internals might. Interpretability tools are increasingly cited by AI labs as evidence for safety claims (for example, checking whether a model harbors a “hidden” objective it wasn’t trained to reveal), and regulators evaluating high-risk AI systems, such as under the EU AI Act, increasingly expect some account of how a system’s outputs relate to its internal reasoning, not just external testing of outputs.
Interpretability is closely related to, but distinct from, the broader field of explainable AI (XAI): interpretability generally refers to understanding how the whole system works, while explainability is about accounting for one specific decision. In practice the terms overlap heavily, and both push in the same direction — turning opaque systems into ones whose behavior can be checked rather than merely trusted.
Exploring it yourself
Interpretability research is unusually open for a fast-moving AI subfield. Neuronpedia, a free, open-source platform, lets anyone browse the internal features and circuits that researchers have already mapped across dozens of models from labs including Anthropic, Google DeepMind, and OpenAI, search millions of activations, and even test how nudging a specific internal feature changes a model’s output — no research background required to click around and get a feel for what a “feature” or “circuit” actually looks like.
In the news
Anthropic’s J-lens research is a live example of interpretability work in action: see our report on how J-lens revealed a hidden workspace inside Claude.
FAQ
Is interpretability the same as explainable AI (XAI)? They overlap heavily. Interpretability usually refers to understanding how a system works internally; explainability usually refers to accounting for one specific output. Many researchers use the terms loosely and interchangeably.
Can interpretability stop an AI from hallucinating or lying? Not by itself. It can reveal, after the fact, when a model’s stated reasoning doesn’t match what actually produced its answer, and it can surface internal signals — like whether a model appears to “know” a fact it declines to state. That is diagnostic information, not a fix on its own.
Do I need to be a machine-learning researcher to use these tools? No. Platforms like Neuronpedia are built for browsing, not just for specialists — though building new interpretability techniques does require research expertise.
Why are AI companies investing in this now? As models are deployed in higher-stakes settings — finance, healthcare, government — labs and regulators increasingly want evidence that a model’s behavior can be understood and predicted, not just that it scores well on tests.
Sources: Anthropic — Tracing the thoughts of a large language model; Wikipedia — Mechanistic interpretability; Wikipedia — Explainable artificial intelligence; Neuronpedia.