A token is the small chunk of text a large language model actually reads and writes — not a whole word, but a piece of one: a common word, a syllable, a punctuation mark, or a single character, depending on how the text breaks down. “Tokenization” is the process of splitting text into these chunks before a model can process it, and it now sits at the center of two things: how AI is billed, and how well it handles different languages.
From text to tokens
Language models don’t read letters or words directly — they process a fixed vocabulary of numbered tokens, typically tens of thousands to over a hundred thousand entries. A tokenizer breaks input text into pieces that match entries in that vocabulary, then converts each piece into a number the model can compute with. The word “tokenization” itself might split into two tokens (“token” and “ization”), while a common short word like “the” is usually its own single token.
Most modern tokenizers build that vocabulary using byte pair encoding (BPE): starting from individual characters, the algorithm repeatedly merges the most frequently occurring pairs into new tokens, so common patterns in the training data collapse into single units while rare ones stay split into smaller pieces. Because the vocabulary is built from whatever text dominates training data, common English words tend to become efficient single tokens, while less common patterns — including entire other languages — get chopped into more, smaller tokens.
Why AI pricing is quoted “per million tokens”
Every token a model reads or generates costs the provider compute, so token count is now the standard billing unit for AI APIs, typically quoted as a price per million tokens processed. It’s also what usage-based AI pricing meters when providers move away from flat subscriptions — the more tokens a task consumes, the more it costs, regardless of how “long” the task feels to a human. A short question might use a few hundred tokens; a long document or a multi-step AI agent task can use hundreds of thousands.
Token count also sets a hard technical limit: every model has a fixed context window — the maximum number of tokens it can hold in a single conversation, including everything read in and generated out. Once a conversation’s token count exceeds that window, earlier content has to be dropped or summarized for the model to keep working.
Not every language tokenizes the same
Because tokenizer vocabularies are shaped by their training data, which is dominated by English and a handful of other widely used languages, the same text translated into different languages can produce very different token counts. A widely cited 2023 study found these differences reach up to 15 times between languages, with speakers of some languages ending up paying at least 2.5 times more than English speakers for equivalent content, while fitting proportionally less of it into the same context window.
Georgian is a documented example. A 2024 study found that an early version of Mistral’s tokenizer, which was missing several letters of the Georgian alphabet and fell back to encoding them byte-by-byte, needed an average of 7.6 tokens per Georgian word — far above the roughly 1.2–1.4 tokens per word typical for English. After the researchers extended the tokenizer’s vocabulary with about 5,500 Georgian-specific tokens, that fell to 2.7 tokens per word, nearly a threefold improvement. Providers with more complete, Georgian-aware vocabularies handle this better by default today, but the underlying pattern — that some languages cost more to process purely because of how their script and grammar tokenize — still holds across most frontier models.
Checking your own token count
Most providers offer a way to see this directly. OpenAI publishes a free tokenizer tool where pasting in any text shows exactly how it splits into tokens and how many there are, no account required. Anthropic doesn’t publish an open tokenizer the way OpenAI does, but offers a token-counting API endpoint so developers can check a prompt’s exact token cost before sending it. Comparing the same sentence in English and in Georgian on either tool makes the disparity concrete.
In the news
The shift toward token-based billing is playing out now: Anthropic recently moved Claude Fable 5 from subscription inclusion to metered, per-token credits, pricing it by exactly the input and output token counts described above.
FAQ
Is a token the same as a word?
No. A token is often smaller than a word — common words may be one token, while longer or rarer words split into several. As a rough rule of thumb, one token is often about three-quarters of an English word, but the proportion varies a lot by language.
Why do different AI models give different token counts for the same text?
Each model family uses its own tokenizer and vocabulary, trained on its own data, so the same sentence can split into a different number of tokens depending on the model.
Does a larger context window mean I get more tokens for free?
No — context window size and price are separate. A larger context window lets a conversation hold more tokens before older content must be dropped, but every token processed is still billed at the same per-token rate.
Can I reduce how many tokens my text uses?
Shorter, more direct prompts use fewer tokens, and cutting repeated or unnecessary context helps. For non-English text, token count is largely fixed by the tokenizer’s vocabulary and can’t be reduced much through rewording alone.