What's the difference between prefill and decode?

Prefill reads and processes the whole prompt at once, which is compute-heavy work suited to parallel processing. Decode then generates the reply one token at a time, which is memory-heavy because each step re-reads the growing KV cache. Disaggregated inference runs each stage on hardware built for its specific demands.

Is disaggregated inference only used by AMD and Cerebras?

No. Nvidia’s Dynamo serving software popularized the same prefill/decode split across pools of its own GPUs, and China’s DeepSeek has said its production models used the technique even earlier. What’s new about the AMD-Cerebras system is splitting the work across two different chip designs from two different companies.

Will this make chatbots faster for me personally?

The main goal is data-center efficiency, not the raw speed of a single reply. Serving more users per watt of electricity can eventually show up as lower subscription or API prices rather than a noticeable speed boost in everyday chats.

What makes a wafer-scale chip different from a regular GPU?

A wafer-scale chip is built from an entire silicon wafer instead of being cut into hundreds of smaller dies. Cerebras is the best-known maker of these; the design packs far more memory close to the compute cores, which is why it suits memory-bound work like the decode stage.

When will the AMD-Cerebras system be available to customers?

The companies plan to deploy it first inside Cerebras’ own data centers, with the combined system reaching commercial customers through Cerebras Cloud in the second half of 2026.

What Is Disaggregated Inference — and How Does It Speed Up AI?

Every time an AI chatbot answers a question, it goes through two very different jobs: reading and understanding the prompt, then generating the reply one word at a time. Disaggregated inference is the practice of running those two jobs on separate, specialized hardware instead of forcing one chip to do both — because the two jobs strain a computer in opposite ways, and no single chip is equally good at both.

The two halves of a single answer

Generating an AI response happens in two stages, usually called prefill and decode.

Prefill is what happens the moment a prompt lands: the model reads the entire input — the question, any attached documents, the conversation history — and converts it into an internal numerical representation, building what is called a key-value (KV) cache along the way. This step can be heavily parallelized, since the whole prompt is available at once, so it mostly needs raw computing power.

Decode is what happens next: the model produces the reply one token at a time, and each new token depends on the one before it, so the process can’t be parallelized the same way. Instead, it depends on repeatedly pulling the growing KV cache in and out of memory, so it’s bottlenecked by how fast data can move, not how many calculations a chip can do at once.

Put simply: prefill is compute-bound, decode is memory-bound. Running both on the same GPU means the chip is over-built for one stage and under-built for the other, at every step.

Splitting the work across chips

Disaggregated inference assigns prefill and decode to separate pools of hardware, each tuned to its job, and shuttles the KV cache between them over a fast interconnect. Nvidia popularized the approach industry-wide with its Dynamo serving software, and Chinese lab DeepSeek has said publicly that its production DeepSeek-V3 and R1 models have run prefill and decode on separate GPU clusters since before most open-source tools supported it.

What’s new is doing this split across two different kinds of AI chips from two different companies. On July 23, AMD and Cerebras announced a joint system that hands prefill to AMD’s Helios server racks — built around its Instinct GPUs — while Cerebras’ wafer-scale engine, a chip cut from an entire silicon wafer instead of hundreds of separate dies, handles decode. Because Cerebras’ giant chip keeps a model’s weights on the chip itself rather than in separate memory modules, it can serve the memory-hungry decode stage with unusually low latency. AMD and Cerebras say the pairing delivers up to five times more tokens generated per watt than running the workload on Cerebras hardware alone.

The trade-off

Disaggregation isn’t free. Splitting a single request across two machines means the KV cache has to move over the network fast enough that the delay doesn’t cancel out the speed gained — which is why these systems rely on high-bandwidth interconnects rather than ordinary networking, and why the approach only pays off at the scale of a data center serving millions of requests, not a single desktop GPU.

Why it matters

For most of the AI boom, the expensive part was training new models. That is changing: running already-trained models to answer billions of daily queries — inference — is now a bigger and faster-growing cost for AI companies than training itself. Every efficiency gain in serving inference translates directly into either lower prices for users or fatter margins for the companies selling AI access, which is why chipmakers keep pushing new hardware combinations to chase it.

Disaggregation also reshapes competition in AI hardware. It weakens the case for buying one type of chip to do everything, and opens room for specialists — a wafer-scale chip that’s excellent at decode, say, or a chip tuned purely for prefill — to win a slice of the inference market even without matching a general-purpose GPU on every metric. That’s part of why AMD and Cerebras are positioning their joint system as an alternative to Nvidia, which still supplies most chips used for both training and inference today.

In the news

AMD and Cerebras detailed their joint inference system, and the performance numbers behind it, in a July announcement that AMD called one of the largest infrastructure opportunities the industry has seen.

The two halves of a single answer

Splitting the work across chips

The trade-off

Why it matters

In the news

Frequently asked questions