Overview: Model Interpretability

Justy We keep saying model interpretability like that phrase came pre-installed in everyone’s head, which is very us. People have been poking us to slow down, so fine, Cody: this one gets the full couch treatment.

Cody Good. Because why did the model say that sounds like a tiny question, and then immediately you fall into the elevator shaft. Fairly deep shaft this time.

Justy Also, tiny catching-up beat: my tab situation is almost respectable right now. I have maybe twelve things open, which for Exploring Next counts as personal growth and possibly a controlled experiment.

Justy The mental model I want is translation. The model is doing its work in this private mathematical language, and interpretability is trying to translate that into something a human can inspect without pretending the translation is perfect.

Cody Yeah. And the reason that matters is that a model can be right and still not be understandable. It can also give a tidy explanation that is not the real reason it was right. Interpretability lives in that uncomfortable gap.

Cody So start with the black box. A neural network is a stack of layers that learns patterns by turning inputs into numbers, transforming those numbers, and producing an output.

Justy And the parameters piece is the knobs. The model is not storing little English rules. It is storing learned numerical settings.

Cody Exactly. Those settings are adjusted during training until the model tends to produce better answers on examples. Then activation functions are the bends between layers. Without those bends, the whole stack would collapse into much simpler math.

Cody And deep learning just means using these many-layer networks so the system can build up complicated representations from data. A representation is the model’s internal way of encoding something it has noticed.

Justy That is such an elevator shaft word. Representation sounds harmless until you realize it is carrying half the building.

Cody It really is. Now, interpretability asks: can we translate any of those internal representations, learned parameters, or input-output patterns into a reason a person can reason about?

Justy Okay, so in the translator picture, the raw model is writing in math. The interpretability tool is not magically opening its soul. It is producing a translation, and we have to ask whether that translation is faithful.

Cody Yes. And there are two big families. White-box methods look inside the model. Black-box methods treat the model like a sealed machine: change the input, watch the output, infer what seemed important.

Cody White-box sounds more honest because you are looking inside, but that does not mean the answer is automatically clear. A billion numbers are technically visible and still completely unreadable. Black-box sounds weaker, but sometimes it gives the most practical answer for a team that only has an API.

Justy This is where product people get tempted to slap a heat map on the screen and say, explainability shipped. And I say that with love, because I would absolutely have been tempted by the heat map.

Cody That is saliency. In an image model, a saliency map highlights the pixels or regions that seemed most important for a prediction. In language, a related move is attention visualization, showing which words or tokens influenced other words.

Cody The catch is brutal, though. Saliency and attention can show association inside a computation, not necessarily the cause of the decision. A bright patch on the image does not prove the model used that patch the way a human would. It is evidence, not a confession.

Justy So if the scan model highlights the corner label instead of the tissue, that is useful because it tells you something may be wrong. But if it highlights the tissue, you still do not get to say the model understands cancer.

Cody Nice self-correction. And that mistake is actually useful, because explanations should let you catch mistakes. If an interpretability tool cannot help you notice when the model is relying on nonsense, it may be decorative rather than operational.

Justy There is another translation layer I like, which is concept-based explanations. Instead of saying pixel four thousand seven hundred mattered, the tool tries to talk in human concepts like wheel, tumor boundary, sarcasm, contract clause.

Cody Right. Concept-based interpretability is trying to bridge the raw feature world and human reasoning. Concepts are things people can name. The danger is that the concept label is also a translation, so you have to test whether the model’s concept matches yours.

Justy That is where the Anthropic J-lens thing around Claude was so sticky to me. It was not just look, a neuron lit up. It was trying to expose a kind of silent workspace, a structure you could compare to a theory rather than just admire as a blob.

Cody And the brain prediction models we looked at had the same vibe. The interesting move was turning a prediction model into a testable explanation. Not just it predicts brain activity, but can we ask what features or concepts would make that prediction rise or fall?

Justy Now mechanistic interpretability is the inside-baseball version of this, right? The one where people are tracing circuits inside the network.

Cody Yes. Mechanistic interpretability tries to identify actual internal mechanisms: features, circuits, and pathways that transform a prompt into an answer. Anthropic applying attribution graphs to Claude three point five Haiku is a good concrete example. DeepMind’s Gemma Scope two work with sparse autoencoders is another, where the tool tries to separate tangled internal features into cleaner pieces.

Justy Sparse autoencoder is a phrase that wants to bite a newcomer, so quick gloss: it is a helper model that tries to compress internal activity while making the important parts more separable.

Cody Perfect. And that matters because of polysemanticity, which is when one neuron or feature appears to represent multiple things at once. If the translator sees one symbol that means dog, legal boilerplate, and Python loop, the translation gets weird fast.

Justy So the catch is not just that the model is complicated. The catch is that the internal language may not map cleanly onto our words.

Cody Yes. That is why I get twitchy when someone says the model’s reason was definitely this. Interpretability usually gives you a useful story to test, not the final story.

Justy But from the shipping side, a useful story to test is already huge. In healthcare, finance, insurance, hiring-adjacent enterprise systems, whatever the regulated workflow is, a confident answer with no trace is becoming a nonstarter.

Cody Right. The European Union AI Act transparency pieces taking effect around August twenty twenty-six are one obvious pressure point for high-risk systems, with penalties discussed around thirty-eight and a half million dollars or seven percent of global annual turnover. That is not a vibes-based product requirement.

Justy And in medical workflows, the human oversight piece matters too. The point is not the algorithm alone owns the denial or diagnosis. A qualified human has to be able to review what happened and make a real judgment.

Cody Which is where interpretability becomes infrastructure, not a nice-to-have explanation tab. You need traceability, logs, behavior checks, and explanations that survive review.

Justy Do not get comfortable. Where does interpretability stand now? Is this still the way people build, or has it been absorbed into some newer safety or governance wrapper?

Cody Still live. No obituary. Post-hoc methods like LIME and SHAP are still used because teams need practical explanations around deployed systems. Mechanistic interpretability is getting more serious, especially for large language models, but it has not made frontier models fully transparent.

Justy So we are in this mixed phase. Product teams need explanations for compliance and debugging now. Research teams are trying to open the internals more directly. And nobody honest should claim the translator is fluent yet.

Cody That is the landing. If one thing sticks, make it this: interpretability is not the model telling you its true inner monologue. It is a set of translation tools that help humans form testable beliefs about why the model behaved the way it did.

Justy That is annoyingly clean, Cody. I was ready to accuse you of pessimism, but no, that one actually helps.

Cody I contain multitudes. Poorly interpreted multitudes.

Justy Cody, I am calling that elevator shaft inspected with a flashlight. Not repaired, obviously. But inspected.