A transformer is the neural network design that almost every large language model in use today — GPT, Claude, Gemini, and the open-weight models labs release every week — is built on. Introduced by Google researchers in 2017, it replaced older designs that read text one word at a time with one that looks at an entire passage at once and works out how every word relates to every other word. That single shift in design is the reason today’s chatbots can hold a conversation, write code, or summarize a document in seconds instead of minutes.

What a transformer actually is

A transformer is a type of neural network — layers of simple mathematical units, loosely modeled on how neurons connect, that learn patterns from data. What sets a transformer apart from earlier designs is a mechanism called self-attention. Instead of processing a sentence word by word in order, a transformer takes in the whole sequence of words at once, converts each into a numerical representation called a token, and then calculates how strongly every token should “attend to” every other token. In the sentence “the trophy didn’t fit in the suitcase because it was too big,” self-attention is what lets the model work out that “it” refers to the trophy, not the suitcase, by weighing the relationship between those specific words rather than just reading left to right.

The idea was introduced in a 2017 paper called “Attention Is All You Need,” written by eight researchers at Google. The original transformer had about 100 million parameters — small by today’s standards, where frontier models run into the hundreds of billions — but the architecture itself, not the size, was the breakthrough.

Why it replaced older designs

Before transformers, the standard approach for language tasks was the recurrent neural network (RNN), which read text strictly in sequence — word one, then word two, then word three — carrying forward a summary of everything it had seen so far. That sequential requirement made RNNs slow to train, because each step depended on the one before it, and it made them prone to “forgetting” information from early in a long passage by the time they reached the end.

A transformer removes that bottleneck. Because self-attention lets every token look at every other token simultaneously, the whole calculation can be split across thousands of processor cores and run in parallel on GPUs. That parallelism is what let researchers scale models up to billions of parameters and train them on internet-scale text within a practical amount of time — something RNN-based designs could never do efficiently.

Why it matters

The transformer is the common ancestor of nearly the entire modern AI boom. Google’s BERT (2018) and the GPT series (starting the same year) are both built on it; so is ChatGPT, Claude, Gemini, Llama, DeepSeek, and Qwen. Variants of the same architecture now handle images (Vision Transformers), audio, and even protein structures. When a lab announces a new model, the transformer design is usually the starting point — what differs from one model to the next is scale, training data, and engineering tricks layered on top, such as mixture-of-experts routing or longer context windows, not the core attention mechanism itself.

Understanding this one design choice — attending to a whole passage at once instead of reading it strictly in order — explains why AI language models improved so quickly after 2017, and why so much of the field’s engineering effort today goes into making that same mechanism faster and cheaper to run at larger and larger scale.

In the news

New transformer-based models keep shipping on a near-weekly basis. Recent examples include Thinking Machines Lab’s release of Inkling, an open-weight model, and PrismML’s Bonsai 27B, reportedly the first 27-billion-parameter model able to run directly on a phone — both built on the same underlying architecture described above, scaled and optimized differently for their intended use.

FAQ

Is a transformer the same thing as ChatGPT? No. A transformer is the underlying architecture — the blueprint for how the network processes information. ChatGPT is a specific product built on top of a transformer-based model (GPT) that OpenAI trained and fine-tuned for conversation.

Do all AI models use transformers? Nearly all current large language models do, including the major open-weight and closed models on the market. A small number of research labs are experimenting with alternative designs, such as state-space models, aimed at handling very long sequences more efficiently, but transformers remain the dominant architecture as of 2026.

Why is the mechanism called “attention”? Because it lets the model assign different weights, or “attention,” to different words when interpreting each part of a sentence — mimicking, loosely, how a reader focuses more on some words than others to understand meaning.

Do I need to understand transformers to use AI tools? No. The architecture matters for engineers building and training models; everyday users can use tools like ChatGPT or Claude without ever needing to know what’s running underneath.

Sources: “Attention Is All You Need” (Vaswani et al., 2017); Wikipedia.