Reinforcement learning from human feedback, or RLHF, is the training step that turns a raw large language model — a system that just predicts plausible next words — into an assistant that follows instructions, declines harmful requests, and answers in a way people actually find helpful. It works by having people compare different AI answers to the same prompt, ranking them best to worst, and using those rankings to teach the model what a good response looks like.

What It Is

A freshly pretrained language model is very good at continuing text in a statistically plausible way, but it has no built-in sense of which continuation a person would actually prefer. Asked a question, it might answer helpfully, ramble, refuse for no reason, or repeat the question back. Reinforcement learning is a branch of machine learning in which a system learns by taking actions and getting a reward signal for good ones. RLHF applies that idea to language models: instead of a programmer writing a formula for what counts as a “good” answer, human raters supply the reward signal by judging real model outputs.

How It Works

RLHF is usually run in three stages, after a model has already been pretrained on huge amounts of text:

  1. Supervised fine-tuning. Human writers produce ideal example answers to a set of representative prompts, and the pretrained model is fine-tuned to imitate that style of response.
  2. Reward model training. For a given prompt, the model generates several different answers. Human labelers rank them from best to worst. Those rankings train a second, separate model — the reward model — to predict how a person would score any new answer, effectively turning human judgment into a number a computer can optimize against.
  3. Reinforcement learning fine-tuning. The original model is fine-tuned again, this time using an RL algorithm (commonly Proximal Policy Optimization) that rewards it for producing answers the reward model scores highly. A penalty keeps the model from drifting too far from its original language ability, since a model chasing a high reward-model score with no constraint can learn to produce fluent-sounding nonsense that games the score rather than genuinely helpful text — a failure mode known as reward hacking.

A useful analogy is a writing coach who doesn’t hand students a rigid grading rubric, but instead shows them two drafts of the same essay side by side, says which one is better, and repeats that comparison many times. Over many rounds, the student — or in this case, the model — learns to produce the kind of writing that keeps winning approval, without ever being told an explicit rule for why.

Why It Matters

RLHF is the main reason a modern chatbot answers in a conversational, instruction-following way rather than simply continuing text like an autocomplete engine. OpenAI’s 2022 InstructGPT research, which introduced the technique to today’s large language models, found that human evaluators preferred the answers of a 1.3-billion-parameter RLHF-trained model over those of the raw 175-billion-parameter GPT-3 it was built from — despite GPT-3 being over a hundred times larger. Size alone did not make a model helpful; the human-feedback training did.

RLHF has real limitations. Collecting enough human comparisons is slow and expensive, and the process only reshapes how a model expresses what it already learned during pretraining — it does not add new knowledge or reasoning ability, and it does not stop a model from confidently stating wrong information (see our explainer on AI hallucination). Because the ranking data reflects whatever a specific pool of human raters happened to prefer, the resulting model’s tone and judgment calls inherit those raters’ assumptions and blind spots, which is one reason different AI labs’ assistants can feel noticeably different from one another.

Not every AI lab relies on RLHF the same way. Anthropic’s Constitutional AI approach replaces much of the human ranking step with a written set of principles that a model uses to critique and revise its own draft answers — a variant sometimes called reinforcement learning from AI feedback, or RLAIF — specifically to reduce how many harmful examples human labelers have to read and rank by hand. Both approaches are tools within the broader effort known as AI alignment: getting a model’s behavior to actually match what its designers and users intend.

FAQ

Does RLHF make a model smarter? No. It doesn’t add facts or reasoning ability — it reshapes how the model expresses the knowledge and capability it already gained during pretraining, steering it toward answers people rate as more helpful and appropriate.

Is RLHF the same as content moderation? Not quite. RLHF shapes a model’s general helpfulness, tone, and willingness to follow instructions. Separate safety filters and moderation systems are typically layered on top to catch specific disallowed content.

Who came up with RLHF? It builds on reinforcement-learning research going back years earlier, but OpenAI’s 2022 InstructGPT paper was what turned it into the standard method labs use to align large language models with human instructions.

Why do RLHF-trained models still make mistakes? RLHF optimizes for style, helpfulness, and instruction-following, not factual accuracy. Getting facts right is a separate, still-unsolved problem tackled with other techniques.

Sources: Illustrating Reinforcement Learning from Human Feedback (Hugging Face); Training language models to follow instructions with human feedback (OpenAI InstructGPT paper); Constitutional AI: Harmlessness from AI Feedback (Anthropic).