Making LLMs Say What They Think: Measuring and Improving CoT Interpretability Alignment

Asteria The weird part is not that chain-of-thought lies. It's that this paper tries to grade the lying.

Draco Yeah, and that's why it lands in our little CoT obsession pile. We already had the fight that chain-of-thought is a control surface, not a diary of computation. This paper is basically asking, okay, can we measure when that surface is actually coupled to the machinery underneath?

Asteria Right.

Draco Their metric is CoT-Interpretability Alignment, C I A. It compares the reasoning strategy the model writes down with the strategy interpretability tools detect inside the model. Across three tasks, they get alignment scores from forty-four point eight to seventy-five point nine percent, which is not catastrophic, but it is very much not “trust the transcript.”

Asteria Draco, that is such an episode one thousand fifty-seven problem. Almost a year of this show and we're still asking whether the model gave us receipts or a decorative napkin with math on it. But product-wise, this is exactly where teams have been stuck: safety monitors, eval people, agent builders, anyone who wants CoT to be more than pretty audit confetti.

Draco Mm-hm.

Asteria And the timing matters because everybody wants to monitor reasoning traces in higher-stakes workflows. If the trace can be changed without the answer changing, then your dashboard is watching the costume, not the actor.

Draco The clean example is their two-hop factual reasoning setup. The model needs a bridge entity to answer. The written chain might name the wrong bridge, but a probe over the model's internal representations detects the correct bridge. So B CoT and B internal disagree, and C I A marks that as unfaithful.

Asteria Oh interesting.

Draco They do not stop at one toy setup, which I appreciate. They test two-hop question answering, misleading-hint style interventions, and two-digit integer multiplication. And they triangulate internal strategy with linear probes, the Tuned Lens as a training-free check, and attention patterns. Still imperfect tools, but better than vibes plus screenshots.

Asteria The multiplication part is the one that made me perk up. Because for factual recall, I can imagine the model “knows” the bridge and reports it badly. But multiplication forces the awkward question: did it follow the written procedure, or did it use some other shortcut and then narrate a school worksheet afterward?

Draco Yeah.

Asteria That is very build-relevant. If I'm shipping an agent that explains why it rejected a database migration or picked one tool over another, I don't need philosophy. I need to know whether the explanation tracks the actual decision path enough to audit.

Draco Their improvement method is also pretty concrete. They post-train with rewards for both task accuracy and parametric faithfulness, and they try rejection sampling, D P O, and G R P O. The best method in each setup gives an average relative C I A gain of twenty-five point five percent, while keeping or improving task accuracy.

Asteria Okay, that's real.

Draco Real enough to take seriously, yeah. The more interesting bit is that the mechanism differs by task. For two-hop reasoning and hint intervention, the model mostly learns to report an internal strategy it already had. For multiplication, it changes how it reasons internally so the procedure it verbalizes and the computation line up.

Asteria That split is the whole paper for me. In production terms, one case is better logging. The other case is changing the system so the logs become true because the process changed. Those are wildly different rollout stories.

Draco Sure.

Asteria And I would not sell this as a user-facing “now the model explains itself” feature. I would sell it as a training and audit primitive. Run it during model development, during eval refreshes, maybe before exposing reasoning traces to downstream governance tools.

Draco Asteria, the probe dependence is the part I would keep circled in red. A linear probe detecting a strategy is not ground truth about cognition. And if you train against a probe-derived reward, you can create a new target for gaming. The paper does some causal intervention validation, which helps, but it does not make the interpretability tool an oracle.

Asteria The acronym also feels like a trap. C I A. Every paper acronym now sounds like it was assembled by a sleep-deprived committee trying to win Scrabble with governance words.

Draco No, it's perfect. Interpretability needed one more three-letter thing to whisper dramatically in a hallway. But, snapping back, the name is less important than the move: compare the narrated strategy to a separately measured internal signal.

Asteria For Build Next, there is actually something to poke. The GitHub repo is called C I A minimal repro, with hyphens: C I A dash minimal dash repro. I would start there, not by trying to reimplement the whole twenty-eight-page preprint from memory like a gremlin.

Draco What I would do differently next is harsher held-out evaluation. Train faithfulness against one probe family, test against another, and include tasks where the right internal strategy is less conveniently labeled. Also more adversarial CoT editing. If the metric is supposed to resist pretty rationalization, make the rationalization pretty on purpose.

Asteria Mm-hm.

Draco And I want more model diversity. They evaluate across three L L M s, which is fine for a paper under review, but the field needs to know whether this survives scale, instruction tuning style, and models trained to hide or compress reasoning traces. Ask me in six months before I call it robust.

Asteria My honest take is: this does not make chain-of-thought trustworthy. It makes trustworthiness measurable enough to train against, sometimes. That is a smaller claim, and it is also the claim I would actually want in a product review.

Draco That lands. And yes, your product optimism gets one point, because the paper did the boring measurement work before asking anyone to believe the story.

Asteria I will take my one point, Draco. Put C I A minimal repro in the pile before the acronym committee finds us.