Overview: Reinforcement Learning from Human Feedback

Asteria Okay, I need to say the embarrassing thing first. We throw around RLHF constantly, and I’m not sure we’ve ever actually defined it cleanly.

Draco Yeah, no, that’s fair. We’ve used it like everybody already knows the shape of it, which is very on-brand for us and also kind of annoying.

Asteria Very Exploring Next of us. But this one keeps showing up in everything from chat models to code tools, so I want the actual machine in my head, not just the acronym.

Draco Good. Because the machine is simpler than the acronym makes it sound. The basic idea is that you take a model that already knows how to talk, then use human judgment to push it toward outputs people actually prefer.

Asteria So think of it like a restaurant with a very talented chef who can make a technically perfect dish and still somehow serve you a plate that’s weirdly off. RLHF is the part where the diners keep saying, no, that one’s better, and the kitchen learns the taste.

Draco Right, and the important bit is that the diners are not rewriting the whole recipe by hand. They’re comparing outputs, and the system learns from those comparisons. That’s why this sits after the model already has general language ability.

Asteria Okay, so before we touch RLHF, the model already has to be able to do the basic thing. That usually comes from supervised learning, which is just training on labeled examples, like prompt and ideal answer pairs.

Draco Exactly. Supervised learning is the model being shown the right answer directly. RLHF comes later, when the question stops being ‘can it imitate?’ and becomes ‘can it behave the way humans actually want in messy cases?’

Asteria And the reason that matters is that ‘what humans want’ is not always one crisp label. Tone, helpfulness, whether the answer is evasive, whether it’s too long, whether it sounds confident when it shouldn’t — that stuff is hard to write down as rules.

Draco Yeah. A next-word model can be fluent and still miss the social contract. It can sound plausible while being misleading, or be correct but obnoxious, or be safe in a way that’s useless. RLHF is trying to steer that whole behavior layer.

Asteria So in our restaurant picture, the chef is the base model, and the diners are human annotators comparing dishes. But how do their opinions turn into training data that a machine can use?

Draco They don’t usually grade every dish from scratch. They compare pairs, or rank a few responses to the same prompt. That preference data becomes a dataset of ‘humans liked A better than B’ over and over, which is much easier than asking for perfect scores on everything.

Asteria Mm-hm.

Draco Then a separate model, called a reward model, gets trained on those comparisons. Reward just means a score the system tries to maximize. In plain language, the reward model is the machine’s guess about what humans would prefer if they saw this output.

Asteria So it’s a taste translator. Human judgment goes in at the edges, and the reward model learns to approximate that judgment at scale.

Draco Exactly. And that proxy step is the whole trick and also the whole danger. You can’t have humans compare every possible answer forever, so you learn a stand-in. But any stand-in can drift, and if the model learns to game the stand-in, you get reward hacking instead of real alignment.

Asteria Wait, let me make sure I’ve got the middle of it. The reward model is not the language model itself. It’s a separate model that looks at an answer and says, basically, this seems more like the kind of thing people would pick.

Draco Yep. Separate model, separate job. And then the original language model gets tuned using reinforcement learning, which is the part where an agent learns by trial and error toward higher reward.

Asteria Reinforcement learning, in the simplest version, is just that loop: try something, get a score, adjust so the better-scoring moves happen more often. No mystery, just a feedback loop with a target.

Draco Right. In RLHF, the score usually comes from the reward model, not from a human clicking every time. The language model generates candidates, the reward model scores them, and the training algorithm nudges the language model toward outputs that score higher.

Asteria That’s the part people hand-wave, and I think it matters. It’s not ‘we asked humans what they like once and the model became wise.’ It’s ‘we built a machine that approximates human preference, then optimized against that approximation.’

Draco And the optimizer usually stays close to the original model so it doesn’t go completely feral. A lot of RLHF setups use PPO, which is Proximal Policy Optimization — a reinforcement learning method that makes small, controlled updates instead of giant jumps.

Asteria Okay, that phrase is doing a lot of work, but I get the point. It’s less like replacing the model and more like nudging it with a leash on so it doesn’t forget how to speak.

Draco That’s actually a decent picture. The leash matters because if you optimize too hard against the reward model, you can get weird outputs that score well numerically and feel wrong to people.

Asteria Which is very on theme for every machine-learning system ever. Optimize the proxy too hard and the proxy becomes the product.

Draco Yeah, that is extremely us. And it’s why the proxy step is where a lot of the real research lives now, because the system is only as good as the signal you can trust.

Asteria Let’s anchor this in actual systems, because otherwise this turns into a floating diagram. We’ve looked at things like ChatGPT and Claude, where RLHF is part of why the model feels more conversational and less like a raw autocomplete engine.

Draco Mm-hm.

Asteria And we also went through that piece on reinforcement learning with metacognitive feedback, where the point was to get the model to express uncertainty more faithfully. That’s a very RLHF-shaped idea: not just ‘be correct,’ but ‘behave in the way humans want around uncertainty.’

Draco Right, and that example is nice because it shows the preference signal can be about more than politeness. It can be about calibration, structure, and whether the model knows when it doesn’t know.

Asteria We also looked at OpenRL, which is a self-hosted post-training API. That’s such a good product example because it turns the whole RLHF-ish workflow into something a team can actually run, instead of treating it like magic dust from the frontier labs.

Draco Yeah, and that’s where the infrastructure story gets real. Once you say ‘post-training,’ you’re talking about the stage after the base model exists, when you’re shaping behavior for a specific use case. That’s the part teams can own, and it’s the part people often underestimate.

Asteria We really do love a system that makes the annoying middle visible. I’m still thinking about the taste model, though. If the reward model is learned from human comparisons, what exactly is it learning to predict?

Draco It’s learning a score that correlates with human preference. Not truth, not goodness in some cosmic sense, just preference as expressed by the annotators in the dataset. That’s why people say RLHF aligns to the preferences of its raters, which is useful but not identical to aligning to everyone.

Asteria That feels important. Because if the annotators prefer a certain tone or style, the model can become excellent at that tone and still miss what a different user would have wanted.

Draco Exactly. That’s the limit people sometimes gloss over. RLHF can make a model more useful and safer, but it’s still a preference system trained on a slice of human judgment. It doesn’t solve the whole alignment problem.

Asteria And the preference data itself sounds expensive. You need trained annotators, you need them to agree often enough, and you need a lot of comparisons.

Draco Yep. That’s the bottleneck. Human feedback is slow and costly, and disagreement is not a bug there, it’s the actual texture of the problem. Two people may prefer different answers for perfectly sensible reasons.

Asteria Which is very annoying and also very human. I can see why teams keep trying to compress this into cheaper variants.

Draco Right, and that’s where alternatives like direct preference optimization, or D P O, come in. It skips the explicit reward model step and tries to learn directly from the pairwise preferences. Different trade-off, less moving machinery.

Asteria So D P O is basically the ‘we like the same goal, we just don’t want the extra translator model’ version?

Draco That’s a decent shorthand, yeah. It still uses preference data, but it changes the training path. The point is that RLHF became a family of methods, not one frozen recipe.

Asteria Okay, that’s useful. And it explains why people keep arguing about whether a system is ‘really RLHF’ or more like preference tuning. The family resemblance is there, but the implementation can change a lot.

Draco Exactly. And there are also approaches like constitutional AI, where the model critiques its own outputs against a set of principles instead of relying entirely on human comparisons. That’s another way of trying to reduce the human labeling burden.

Asteria But then you’re moving the judgment into a rule set, right? Which sounds cleaner until the rules get vague or you realize the model is just good at sounding compliant.

Draco Yeah. Every shortcut introduces its own failure mode. RLHF’s failure mode is that the reward model becomes the thing you optimize instead of the thing humans meant. Constitutional approaches have their own version of that, where the model learns the shape of the constitution more than the spirit.

Asteria The model learning the vibes of the constitution is such a cursed sentence, but I know exactly what you mean.

Draco That’s the problem in one line, honestly. And this is why I don’t love people saying RLHF solved alignment. It solved some product problems very well. It did not solve the philosophical one.

Draco Yeah. And if that stuck, then the rest of the acronym stops being ceremonial.

Asteria Good, because I cannot believe we’ve spent this long finally saying it plainly on Exploring Next. Anyway, that’s enough acronym cleanup for one Wednesday.