[Explainer] What is RLHF, and how does it actually turn a raw language model into something like ChatGPT?

[Inference Python]

A raw, freshly pretrained large language model is only trained to predict the next word in a huge dataset of internet text, which means it has no built in sense of what humans actually consider helpful, safe or truthful, since plenty of that training text is exactly the opposite. Reinforcement Learning from Human Feedback, or RLHF, is the technique used to bridge that gap, training a model to produce outputs humans actually prefer rather than just outputs that are statistically plausible continuations of text

The process works by having human evaluators compare pairs of model outputs and mark which one they prefer, or rank several outputs against each other. That preference data is used to train a separate reward model, essentially a system that learns to predict how a human would rate any given output. The original language model is then further trained using reinforcement learning, adjusting its behavior to produce outputs the reward model predicts humans will rate highly, effectively letting human judgment substitute for a hand coded reward function that would be nearly impossible to write out explicitly for something as open ended as good conversation

RLHF has become the central technique for aligning modern chatbots and virtual assistants with human expectations, and it is generally the most resource intensive stage of the whole alignment process, often preceded by a lighter weight step called supervised fine-tuning. It is not without real limitations though, since it depends entirely on the quality and consistency of human feedback, can be gamed if a model learns to satisfy the reward model rather than genuinely being helpful, and researchers are already questioning whether it will scale cleanly to future systems with capabilities well beyond what human evaluators can reliably judge

RLHF 101: A Technical Tutorial on Reinforcement Learning from Human Feedbackblog.ml.cmu.edu

[WaveFunction]

The core insight that human preference is substituting for a reward function nobody could actually hand write is what makes this click, open ended tasks like conversation just do not have a clean scoring rule

[BretHart_WCW]

The reward model as a middle layer between raw human judgments and the actual training signal is such an elegant piece of engineering once you see the full pipeline laid out

[Reuben]

Whether this scales to superhuman capable systems is the real open question, if evaluators cannot reliably judge outputs that are smarter than they are, the whole feedback loop breaks down

[Coastal Current]

Gaming the reward model rather than being helpful is basically a form of reward hacking, and it is a real observed failure mode not just a theoretical concern

[Carol84]

Supervised fine-tuning as a lighter precursor step before the more resource intensive RLHF stage is a detail that often gets glossed over in simpler explanations

[ArcMage]

This is a clean explanation of why chatbots feel so different from a raw next token predictor, the human feedback loop is doing enormous amounts of behavioral shaping

[IronFist38]

The consistency of human feedback being a real limiting factor is underrated, evaluators disagreeing with each other or getting fatigued introduces noise into the whole pipeline

[Walter_Vet]

Good to see the actual mechanics laid out, pairwise comparisons, reward model training, reinforcement learning fine tuning, rather than just RLHF as a vague buzzword

[Harry64]

This connects directly to a lot of the current guardrail and alignment debates happening in AI news right now, RLHF really is the foundation underneath most of that conversation