What is a token in AI, and what is tokenization?

What a token is

A token is the smallest unit of text that a language model works with. Rather than reading letters one by one or whole documents at once, a model operates on a sequence of tokens, chunks of text that the system has decided to treat as atomic. Depending on the word, a token can be an entire common word ("the"), a fragment of a longer or rarer word ("token" plus "ization"), a single character, a space, or a punctuation mark.

Crucially, a model never sees the letters directly. Each token is mapped to an integer ID drawn from the model's fixed vocabulary, and those IDs are what get converted into the numeric vectors (embeddings) the network actually computes on. Microsoft's developer documentation describes tokens as "words, character sets, or combinations of words and punctuation" that models use to break down text before processing.

So when people say a model has a certain context window or that an API charges per token, they are referring to this unit, the model's native currency for text, not to words or characters as a human would count them.

How tokenization works

Tokenization is the process of converting a string of text into that sequence of tokens. Early approaches were simple: word-based tokenization splits on spaces and punctuation, and character-based tokenization treats every character as a token. Each has a fatal flaw. Word-based schemes produce enormous vocabularies and choke on words they have never seen (out-of-vocabulary words), while character-based schemes create very long sequences and force the model to learn language from scratch character by character.

Modern large language models therefore use subword tokenization, which strikes a balance. Frequent words are kept as single tokens, while rare or novel words are broken into meaningful sub-pieces the model has seen before. This keeps the vocabulary manageable (typically tens of thousands of tokens) while guaranteeing that any input, including invented words, misspellings, or code, can always be represented by combining smaller pieces. Hugging Face's documentation walks through exactly this progression from word and character tokenizers to subword methods.

The dominant subword algorithm is byte-pair encoding (BPE), used by the GPT family. BPE starts from individual characters (or bytes) and repeatedly merges the most frequently occurring adjacent pair into a new token, building up a vocabulary of common fragments and whole words. Related methods include WordPiece (used by BERT) and Unigram/SentencePiece. Once text is tokenized, each token is looked up as an integer ID before entering the model.

How many tokens is that? The rule of thumb

For English text, a widely used approximation is that one token corresponds to about four characters, or roughly three-quarters of a word. Put the other way, about 100 tokens equal roughly 75 words, and a single page of prose is on the order of 500 tokens. These are estimates, not exact conversions, because token counts depend on the specific words and the specific tokenizer.

Common, whole words tend to be a single token, while long, rare, or technical words split into several. Whitespace and punctuation also consume tokens. The only way to know an exact count is to run the text through the model's actual tokenizer, which is why providers publish tokenizer tools and libraries for this purpose.

The practical point is that token counts and word counts diverge, and the gap widens for unusual text, code, or non-English languages, so estimating a prompt's size in words can be misleading.

Why tokens matter: context windows and pricing

Tokens are the unit in which a model's context window is measured. The context window is the maximum number of tokens, the prompt plus the generated response, that a model can consider at once. A model advertised with, say, a 128,000-token context can hold roughly 96,000 English words in view. When a conversation or document exceeds that budget, earlier content must be dropped or summarized, so tokenization directly determines how much a model can "remember" in a single pass.

Tokens are also the unit of billing. Commercial LLM APIs charge per token, usually with separate rates for input (the tokens you send) and output (the tokens the model generates). This means the same request costs more if it is phrased verbosely or in a language that tokenizes inefficiently, and it makes token-awareness a real lever on the economics of running AI at scale. Trimming prompts, reusing cached context, and choosing efficient formats all reduce token spend.

Because both the capacity limit and the cost are denominated in tokens, understanding tokenization is not an academic detail, it is central to designing prompts, estimating budgets, and deciding what fits in a single call.

The multilingual problem

Tokenizers are trained predominantly on English and other high-resource languages, and this creates a measurable inequity. The same sentence expressed in a language poorly represented in the tokenizer's training, or written in a non-Latin script such as Thai, Burmese, or many Indian languages, is often split into far more tokens than its English equivalent, sometimes several times as many.

This has real consequences. Because context windows and pricing are measured in tokens, speakers of those languages get less usable context and pay more for the same content, an issue documented in the 2023 study "Language Model Tokenizers Introduce Unfairness Between Languages" by Petrov and colleagues, which found large disparities in tokenization length across languages. It is a concrete example of how a low-level engineering choice produces uneven real-world outcomes.

The effect is gradually shrinking as newer tokenizers are trained on more multilingual data and larger vocabularies, but it remains a genuine consideration when building or budgeting AI systems for non-English users.

Why tokenization explains some model quirks

Because a model perceives text as tokens rather than individual letters, some of its odder failures make sense. A classic example is a model miscounting the letters in a word, for instance, insisting on the wrong number of a given letter, because the word may be a single token and the model has no direct view of its constituent characters. Character-level tasks, reversing a string, precise spelling, exact letter counts, are systematically harder for this reason.

Tokenization also contributes to weaknesses in arithmetic and with numbers, since digits can be grouped into tokens in ways that do not align with place value. And identical-looking text can tokenize differently depending on surrounding spaces or capitalization, which occasionally produces surprising behavior.

The broader lesson is that tokenization is not a neutral pre-processing step but a design decision that shapes what a model finds easy or hard. It sits between human-readable text and the numbers a neural network computes on, and its choices ripple through cost, capacity, fairness, and capability.

Frequently asked questions

How many words are in a token?

For English, a common rule of thumb is that one token is roughly three-quarters of a word, or about four characters, so about 100 tokens correspond to roughly 75 words. These are approximations: short common words are usually a single token, while long, rare, or technical words split into several, and punctuation and spaces also count. The only exact way to measure is to run the text through the model's specific tokenizer, and the ratio differs for other languages.

What is the difference between a token and a word?

A word is a human unit of language; a token is the unit a model actually processes, which may be a whole word, part of a word, a single character, or punctuation. Modern models use subword tokenization, so common words are often one token while rarer words are split into several pieces. That is why token counts and word counts diverge, especially for unusual text, code, or non-English languages.

What is byte-pair encoding (BPE)?

Byte-pair encoding is the most common subword tokenization algorithm, used by the GPT family. It begins with individual characters or bytes and repeatedly merges the most frequently occurring adjacent pair into a single new token, gradually building a vocabulary of common fragments and whole words. The result keeps frequent words as single tokens while still being able to represent any rare or novel word by combining smaller pieces, which avoids out-of-vocabulary failures.

Why does tokenization affect the cost of using an LLM?

Commercial LLM APIs charge per token, typically with separate rates for input and output tokens. Because tokenization determines how many tokens a given piece of text becomes, verbose phrasing or an inefficiently tokenized language increases the token count and therefore the cost of the same request. Token counts also determine how much fits in the context window, so managing tokens affects both what is possible in one call and what it costs.

Why do some languages use more tokens than English?

Because tokenizers are trained mostly on English and other high-resource languages, so their vocabularies represent English efficiently. Text in under-represented languages, or in non-Latin scripts, often gets split into many more tokens for the same meaning. Since context windows and pricing are measured in tokens, this means such users get less usable context and pay more, a disparity documented by Petrov et al. (2023). Newer, more multilingual tokenizers are narrowing the gap.

Why do language models miscount letters in a word?

Because a model sees text as tokens, not individual characters. A word may be represented as a single token, so the model has no direct view of its constituent letters and can get character-level questions, like how many times a given letter appears, wrong. The same tokenization effect makes exact spelling, string reversal, and some arithmetic harder, since the model's native unit is the token rather than the character or digit.

Related

Explainers: LLM, GenAI, Diffusion model

Glossary: Context window, Embedding (vector), Transformer (architecture)

Sources

• IBM Think — What is tokenization?
• Hugging Face — Summary of the tokenizers
• Microsoft Learn — Understand tokens
• Petrov et al. (2023) — Language Model Tokenizers Introduce Unfairness Between Languages
• Google — Machine Learning Glossary