When is Your LLM Steerable?
A new paper investigates whether you can predict if a steering attempt will succeed by looking at the model's hidden states from just the first few tokens—before decoding the full response. The authors build ASTEER, a dataset of 1.4M steered generations labeled for success/failure across 150 concepts, then train a GBDT classifier on early hidden-state features to predict three outcomes: under-steer, success, or over-steer. The predictor hits ~0.7 macro-F1 on unseen concepts and cuts the cost of finding good steering strengths without expensive full rollouts.
Transcript
Justy Okay so I was reading this paper on steering and it just… it immediately made me think of the mess we've been dealing with in production.
Cody Yeah?
Justy Right now if you want to steer a model — like, inject a direction into the hidden states to push it toward some behavior — you have to try a steering strength, generate the whole thing, call a judge model, see if it worked, and if not, try again. It's… it's expensive.
Cody And the same steering coefficient just doesn't work across different prompts or concepts.
Justy Exactly. So this paper — 'When is Your LLM Steerable?' — they're basically asking: can you tell if steering is going to work before you finish decoding? Like, from the first few tokens.
Cody That's interesting. So the hypothesis is that early hidden states encode information about whether the intervention will eventually succeed.
Justy Right. And there's precedent for this — other work has shown that hidden states early on predict hallucination, correctness, harmfulness. So why not steering efficacy?
Cody Okay, that's actually a solid framing.
Justy So to test it, they built this massive dataset — ASTEER — one point four million steered generations across one hundred fifty concepts. Each one labeled for whether the steering succeeded, under-steered, or over-steered.
Cody Wait, three categories?
Justy Yeah. Under-steer means the concept didn't get incorporated into the response. Over-steer means the model fixated on the concept and broke coherence — couldn't actually answer the prompt anymore. And success is both: you get the concept in the response AND you still answer the question.
Cody That's cleaner than treating it as binary pass-fail.
Justy Because the failure modes are different. You need to know which way you're failing to know how to adjust.
Cody Mm-hm.
Justy So they take those 1.4 million examples and extract features from the hidden states. Not just at the steering layer — they're comparing states across the whole network, layer by layer, and across the first few decoding steps.
Cody So the feature set is basically: 'how does the steering perturbation propagate downward through the layers and forward through time?'
Justy Yes. They're capturing the geometry of how the steering signal moves through the model.
Cody And then they train a classifier on that.
Justy GBDT — Gradient Boosting Decision Trees. Feed it those early hidden-state features and it predicts whether you're going to succeed, under-steer, or over-steer.
Cody How does it generalize?
Justy Around zero-point-seven macro-F-one on unseen concepts. So it's seen one hundred fifty concepts during training, and when they test it on new ones it still hits that score.
Cody Okay, that's… that's actually pretty solid. I would've expected worse drop-off.
Justy Right? And then they use the predictor as a guide for steering-strength search. Instead of grid-searching over coefficients with full rollouts, you use the predictor to narrow down which strengths are even worth trying.
Cody How much cost do you save?
Justy They hit near-optimal performance with a small fraction of decoding cost. So you're not sacrificing quality, you're just… not wasting compute on attempts that the predictor already flagged as likely to fail.
Cody That makes sense. The key insight is that the model's behavior in the first few tokens is structurally informative about the long-term outcome.
Justy Exactly. Like, the hidden states after steering but before full generation already encode whether the intervention took hold.
Cody This is such a 'measure the thing you want to optimize' moment.
Justy Right? We've been throwing away that signal. You steer, you generate, you judge — but you never look at whether the steering actually landed in the first place.
Cody The one thing I'd want to know is whether the predictor is actually learning steering geometry or just picking up on 'this prompt is harder, this concept is harder' in general.
Justy That's fair. The paper shows feature importance — they do some interpretability work showing that the layer-by-layer propagation features actually matter, not just prompt or concept difficulty.
Cody Okay, good. Because if it's just learning 'this concept is always hard to steer toward,' that's less useful for the search application.
Justy From a product angle, this is huge. Right now you're burning tokens on failed steering attempts. This lets you predict failure early and either skip the attempt, adjust the strength, or try a different concept.
Cody Yeah. And if you're running agents that steer models at inference time — which is getting more common — every failed attempt costs real time and compute.
Cody Yeah. That dataset is the thing other people are going to build on.
Justy For sure. Alright, I'm going to stop burning cycles on steering attempts that were doomed from token one.