Stealing Reasoning Traces from Proprietary LLM APIs
Justy and Cody discuss a new paper showing how encrypted reasoning traces from proprietary LLMs can be stolen by replaying them into weaker sibling models from the same provider, enabling distillation, data leaks, and prompt injection. They unpack the attack mechanism, its real-world impact via scraped public logs, and whether mitigations exist, weighing the paper’s claims against their own experience with API security and model guardrails.
Transcript
Justy Okay, this paper’s making the rounds — ‘Stealing Reasoning Traces from Proprietary LLM APIs’ — and honestly, it’s the kind of thing that sounds like a CTF challenge until you realize people are spilling API keys in public GitHub repos because they didn’t know the encrypted blobs were readable.
Cody Yeah, I saw that. The core insight isn’t new — we’ve known these reasoning blocks get sent back and forth for stateless continuity — but the paper shows they’re fully interchangeable across models within the same vendor’s ecosystem. That’s the real kicker.
Justy Right, so they’re saying you can take an encrypted thinking trace from, say, Claude Opus 4.8, toss it into a Haiku 4.5 call, and Haiku will just… decode it and spit out the reasoning in plaintext? No jailbreak needed on the big model?
Cody Exactly. The frontier models have all this alignment training to prevent reasoning leakage, but the smaller, cheaper siblings? They’re optimized for cost, not guardrails. So you use the weak model as a decryption oracle. One API call to get the trace, another to feed it to the small model and say ‘output your reasoning’ — and boom, you’ve got the chain of thought.
Justy That is honestly kind of brilliant in a horrifying way. It’s like using the kid sibling to crack the older one’s diary because they never bothered to lock it.
Cody Yeah, and the paper backs it up across Anthropic, OpenAI, and Google. They show it works for distillation — so you can copy a model’s reasoning style — but then they go further: they scraped 315,320 of these thinking blocks from public logs and found real secrets.
Justy Wait, 315k blocks? And they got what — 367 PII artifacts and 182 credentials? That’s wild. Were people just pasting their session logs into Stack Overflow or something?
Cody Basically. Devs share logs for debugging, not realizing the encrypted chunks contain stuff like API keys, passwords, even personal emails that the model pulled from context or memory but never showed in the final answer. The user sees a safe refusal, but the reasoning trace had the harmful payload all along — and now it’s extractable.
Justy So it’s not just about stealing a model’s secret sauce — it’s about accidental data leaks from users who had no idea they were sharing sensitive info. That feels… worse, in a way. At least distillation is model-vs-model. This is invading privacy.
Cody And the fourth vector they mention — invisible prompt injection. You could poison a reasoning block with malicious instructions, get it logged and shared somewhere, then when someone replays it, the weak model executes the injection without anyone seeing it coming. That’s nasty.
Justy Yeah, especially if it gets baked into some agentic workflow that’s auto-replaying logs. Okay, so mitigations — what do they suggest? Because if the fix is just ‘don’t share logs,’ we’re screwed.
Cody They propose cryptographic bindings — like tying the encrypted trace to a user ID or session so it won’t decode in another context. Or maybe using non-interchangeable formats per model. But honestly, that breaks the statelessness they were going for in the first place.
Justy Which is the trade-off, right? You want zero server-side storage for scaling, but then you have to trust the client not to leak or replay stuff. Feels like we’ve seen this movie before with JWTs or something.
Cody Exactly. Stateless is great until the client becomes the attack surface. I’d want to see if the vendors actually roll out fixes — or if they just rely on obscurity and hope nobody scrapes logs at scale.
Justy Given how many API keys I’ve seen in public repos just from casual googling… yeah, I’m not holding my breath. But hey, at least now we know to tell devs: treat those thinking blocks like they’re plaintext secrets, even if they look like gibberish.
Cody Yeah. And maybe next time we’re evaling a provider, we ask not just ‘how smart is your model?’ but ‘how leaky is your reasoning pipe?’
Justy That’s going in the vendor checklist. Alright, I think that’s enough paranoia for one Wednesday. What’s next on your radar?
Cody Honestly? I’m going to check if any of our old session logs have floating around. Just in case.
Justy Good call. I’ll join you — after we change a few passwords, obviously.