AI model training happens in three distinct stages, each teaching the model something different. Understanding these stages explains both why frontier AI is expensive and why models behave the way they do.

Stage 1: Pre-Training — Learning Language from the Internet

The first stage is called pre-training, and it is by far the largest. The model is shown vast amounts of text — web pages, books, code, scientific papers — and trained to predict the next word in each sequence. This sounds simple, but at the scale of hundreds of billions to trillions of words, something remarkable happens: the model begins to learn grammar, facts, reasoning patterns, and the relationships between ideas, without anyone explicitly programming any of it.

This approach is called self-supervised learning: the training labels come from the data itself — the next word in the sentence — so no human annotation is needed. The model is not being told what to think; it is learning statistical patterns across an enormous slice of recorded human knowledge.

Pre-training is where most of the cost sits. Training a frontier model requires thousands of specialized AI chips running for weeks or months; estimated compute costs for GPT-4-class pre-training ran to roughly $100 million, and newer frontier models are projected to cost several hundred million dollars or more.

The result of pre-training is a base model: a system that can complete text fluently and knows an enormous amount, but has no concept of instructions, questions, or conversation.

Stage 2: Instruction Tuning — Learning to Follow Commands

A base model is not a chatbot. It will complete sentences, but it will not answer questions or follow directions reliably. The second stage, supervised fine-tuning (also called instruction tuning), bridges that gap.

Human trainers write thousands of example conversations: a question or task paired with an ideal response. The model is trained on these curated examples, learning that a request for help with a cover letter should produce a cover letter, not a continuation of the literal sentence. This stage uses far less compute than pre-training, but the quality of the examples matters enormously.

After instruction tuning, the model becomes much more useful — but it still makes mistakes, can be inconsistent, and does not reliably behave the way users prefer.

Stage 3: Alignment — Learning What Humans Actually Want

The third stage — often called RLHF (Reinforcement Learning from Human Feedback) — is what makes the difference between a capable model and one people actually want to use.

The process works in two steps. First, human raters evaluate pairs of model responses to the same prompt, marking which one is better — more helpful, more accurate, safer. These ratings train a separate reward model: a smaller network that learns to predict which responses humans prefer. Second, the main language model is fine-tuned using reinforcement learning to generate responses that score highly on the reward model. It is, in effect, learning from human judgment at scale.

Anthropic published key early work on this approach in 2022. It is the primary technique behind ChatGPT, Claude, Gemini, and most other conversational AI systems. Variants such as Constitutional AI (Anthropic’s approach) and RLAIF (reinforcement learning from AI feedback) reduce reliance on human annotators by using a set of written principles or an AI system to generate preference labels — allowing alignment to scale further.

Alignment is what gives a model its personality: the tendency to be helpful, to decline harmful requests, to admit uncertainty. A model trained without this stage can be knowledgeable but erratic and unreliable. Crucially, a model that copies another model’s capabilities through unauthorized distillation inherits the knowledge — but none of the alignment training, which is why such copies are considered a safety risk as well as an intellectual property issue.

Why Training Costs Billions

Frontier AI training is expensive because all three stages require large amounts of compute, high-quality data, and human time. Pre-training dominates the compute bill; instruction tuning and RLHF dominate the human-labor bill. A team of annotators must review, rank, and write example responses — a bottleneck that no amount of hardware can fully replace.

When an AI company says a model is “in training,” it typically means the pre-training stage is still running. Completing pre-training takes the longest; instruction tuning and alignment follow afterward. The entire pipeline for a frontier model can span many months.

In the News

Meta’s AI chief recently claimed that its in-training model, codenamed Watermelon, has already reached capabilities comparable to OpenAI’s GPT-5.5 — before the model has even finished training. That claim offers a glimpse of how fiercely companies now compete during the training stage itself. Read the full brief: Meta AI Chief Claims In-Training “Watermelon” Model Has Caught Up to GPT-5.5.

FAQ

Do all AI models go through these three stages?
Most modern conversational AI models do, though the exact methods and proportions vary. Smaller or specialized models may skip or abbreviate stages. Open-weight models released for community use sometimes skip the full alignment stage.

What is the difference between training and fine-tuning?
Pre-training is training from scratch on massive data. Fine-tuning starts from an already pre-trained model and continues training on a smaller, more specific dataset. Instruction tuning and RLHF are both forms of fine-tuning.

Can I fine-tune a model myself?
Yes — many open-weight models (such as Meta’s Llama family) can be fine-tuned on consumer or cloud hardware. Instruction tuning a small model for a specific task is within reach of most developers. Pre-training a frontier model from scratch is not — it requires data center–scale compute.

Why do AI models sometimes get worse after updates?
Alignment is an optimization process, and it is possible to overfit — to make a model so cautious or so focused on one metric that its general capabilities degrade. This phenomenon, sometimes called the alignment tax, is an active area of research.

Sources: Reinforcement Learning from Human Feedback — Wikipedia · Anthropic — Training a Helpful and Harmless Assistant with RLHF · Hugging Face — Illustrating RLHF