Ask a frontier model how many r’s are in “strawberry” and, for about two years, you had a decent chance of being told two. It became the standard party trick for anyone arguing that these systems are less clever than they look, and the standard explanation was always the same: the model can’t see letters, it only sees tokens.
That explanation is elegant, widely repeated, and mostly wrong. The research that has actually measured it points somewhere more interesting — and more unsettling. But to get there you need to know what a token is, because tokens do genuinely shape what these systems find easy and what they find absurdly hard. They also quietly decide what you pay, how much fits in the context window, and whether writing in your own language costs you more than writing in English.
What actually arrives at the model
A language model never receives your sentence. It receives a list of numbers.
OpenAI’s own tokeniser library puts it plainly: “Language models don’t see text like you and I, instead they see a sequence of numbers (known as tokens).” [1] Before anything else happens, your text is chopped into pieces and each piece is swapped for its index in a fixed vocabulary. The word strawberry becomes three numbers. That is the entire input.
The pieces are not letters, and they are not words. They are whatever fragments turned out to be statistically common in the text the tokeniser was built on. Roughly four characters each, on average, for English. [1]
One correction worth making early, because most explanations skip it. “The model can’t see letters” is too strong. Modern tokenisers are byte-level and lossless, so the letters are recoverable in principle — and researchers have shown that models do encode which characters a token contains. Probing classifiers trained on token embeddings alone can predict whether a given letter is present, across models including GPT-J, BERT and RoBERTa, and the effect holds in Arabic, Devanagari and Cyrillic scripts. Larger models do it better. [2]
So the honest version is not “the model cannot know the letters.” It is: the model is not shown the letters as separate things, and has to reconstruct that knowledge indirectly. Keep that distinction. The rest of this article depends on it.
Where tokens came from
The algorithm doing the chopping is usually Byte Pair Encoding, and it began life as something else entirely. Philip Gage published it in 1994 as a data compression technique: find the most common adjacent pair of symbols, replace it with a new symbol, repeat. [3]
Eleven years later, Sennrich, Haddow and Birch at Edinburgh borrowed it for a completely different problem. Machine translation systems had fixed vocabularies, but language does not — there is always a word the system has never seen. Rather than giving up on unknown words, they encoded them as sequences of subword pieces. It worked: 1.1 BLEU better on English–German, 1.3 on English–Russian. [4]
That is the whole origin story. A file-compression trick from 1994, repurposed in 2015 to handle rare words, is now the front door to every large language model you use. Nobody designed it to be good at spelling.
The strawberry problem, and why the popular explanation is only half right
Here is what actually happens to the word. These splits are read directly off OpenAI’s own tokeniser tool:
cl100k_base, the tokeniser behind GPT-4 and GPT-3.5, strawberry arrives as str · aw · berry. Under the newer o200k_base it arrives as st · raw · berry. Neither split puts a boundary anywhere near the two r’s people are asking about — both are buried inside berry.raspberry is a single token. So is the plural strawberries — more common in the training text than the singular, so it earned its own slot. Shout it and STRAWBERRY becomes four pieces: STR · AW · B · ERRY. Nothing about how the word looks on the page survives the journey.So far this supports the popular story. Then you look at what happens when researchers actually measure it.
For counting tasks, tokenization properties are weakly correlated with correctness.
That is CharBench, a character-level benchmark two orders of magnitude larger than anything before it, accepted at AAAI-26. Across leading open and proprietary models it found average accuracy of just 43.6% on character-level tasks. But when the authors checked why, the tokenisation explanation did not hold up for counting. What predicted failure was the length of the word and how many of the letter there actually were — not how the tokeniser had cut it. [5]
Tokenisation was not exonerated everywhere. The same paper found that for questions about where a character sits inside a word, performance does get worse as the containing token gets longer — longer tokens genuinely do obscure position. So the split verdict is: tokenisation hurts “which letter is third,” and has little to do with “how many r’s.” Which is exactly the question everyone asks.
A separate group went further, designing evaluations specifically to test the popular conjectures — tokenisation, architecture, training data — and concluded bluntly that “conjectures about inherent deficiency of LLMs” are “invalid.” Their finding was that getting the model to reason through the problem fixed it more reliably than fine-tuning or examples. [6] Which is why reasoning models mostly get strawberry right today, without anyone having touched the tokeniser.
And then, in April 2026, a mechanistic study went looking inside. Probing LLaMA, Qwen and Gemma on questions like “how many p’s are in apple,” the authors found something genuinely strange:
Models often compute the correct answer internally yet fail to express it at the output layer.
The character information is present in the early and middle layers. It is then suppressed by a small number of later components — principally the last two MLP layers, which the authors call “negative circuits.” Their conclusion is that these failures are “not due to missing representations or insufficient scale” but to “structured interference within the model’s computation graph” — and that this is why the errors persist and can get worse with scale and instruction tuning. [7]
Read that again, because it is the most interesting sentence in this article. The model frequently knows the answer and then talks itself out of it.
In fairness, the other side. Not everyone agrees. A paper presented at EMNLP 2025 — titled, inevitably, “The Strawberry Problem” — argues tokenisation is a fundamental limitation, and found that character-level ability emerges suddenly and only late in training. [8] Even there, though, the authors conclude that learning character composition is “not fundamentally different from learning commonsense knowledge” — difficult, but not walled off.
The fair summary: tokenisation makes character-level information awkward to get at. It is not, on the current evidence, the thing that produces the wrong number.
Numbers, and why arithmetic is harder than it should be
Tokenisation has a cleaner, less disputed effect on arithmetic.
Type 1234567890 and the model receives 123 · 456 · 789 · 0. That is not arbitrary: the tokeniser’s own rules cap runs of digits at three and chunk them from the left. [1]
Now consider that addition carries from the right. The groups the model receives are misaligned with the operation it needs to perform. Different model families made different choices here — LLaMA and PaLM went with single-digit tokenisation, while GPT-3.5 and GPT-4 use separate tokens for one-, two- and three-digit numbers — and the choice measurably matters. Forcing right-to-left grouping by comma-separating numbers improves performance, and the errors that remain follow “stereotyped error patterns,” which suggests the model is doing something systematic rather than guessing. [9]
Two encouraging notes from the same work: models can convert between groupings when prompted to work step by step, and the gap between grouping directions shrinks as models get larger.
The bill, and the part of it you cannot see
Tokens are the unit you are charged in and the unit your context window is measured in. Both facts have consequences that are easy to miss.
The rules of thumb OpenAI publishes for English: one token is roughly four characters, or about three-quarters of a word. A hundred tokens is about seventy-five words. [10] Useful for estimating, and worth knowing they are English-specific — more on that shortly.
Two things that surprise people:
- You are billed for text you never see. OpenAI’s own documentation warns that reported output tokens include “all tokens generated by the model, not only the text visible in a response” — including formatting tokens that appear nowhere in the message content. The count you are charged for can exceed the visible text even when reasoning tokens are reported as zero. [11]
- A vendor can change the tokeniser and move your bill without changing the model. Anthropic documents that Claude 4.7 and later use a newer tokeniser under which “the same input text produces approximately 30 percent more tokens than on earlier models,” and explicitly warns against reusing older counts for cost estimates. [12] Nothing about the text changed. Only where the cuts fall.
For scale, context windows as they stand in August 2026: Claude Opus 5, Sonnet 5 and Fable 5 offer one million tokens; [12] GPT-5.6 offers 1.05 million; [13] Gemini 3.7 Flash publishes the exact figure 1,048,576, which is 220. [14] Three vendors, one engineering constant, three different roundings.
The same sentence costs more in some languages
This is the part with real consequences for real people, and it is also where you have to be careful about dates.
In 2023, researchers at Oxford measured the same 2,000 human-translated sentences across seventeen tokenisers and found enormous disparities. On the GPT-4 tokeniser, Shan required 15 times more tokens than English to say the same thing. [15] That figure got quoted everywhere, and it is still being quoted now.
It is out of date, and repeating it as current would be exactly the mistake this site is trying to stop making. The tokenisers changed. A 2026 comparison of the old cl100k_base against the newer o200k_base, on the same corpus, found the penalty for Indian languages fell dramatically: Hindi from 4.08× to 1.34×, Bengali 6.52× to 1.91×, Tamil 9.89× to 2.57×, Malayalam 13.04× to 2.85×. The mean penalty across those languages dropped from about 8× to about 2× — a 73% reduction, with no change to the scripts themselves. [16]
The problem has not gone away, though. Measured on current tokenisers, African languages still carry a median penalty of 1.88×, and the worst cases remain severe — Amharic at 7.36×, N’Ko at 8.92×. Every one of the twenty languages studied carried some premium. [17]
Three things worth understanding about why:
- It is not simply about script. Yoruba is written in the Latin alphabet and still carries one of the higher penalties measured. Arabic script does comparatively well because it is well represented in training data. What drives the penalty is how much of the tokeniser’s vocabulary a language was allocated — which is a function of how much text in that language went in. [17]
- It is not always a penalty. On several multilingual tokenisers, Chinese comes in below English — fewer tokens for the same meaning. [15]
- It compounds. More tokens means a bigger bill and less room in the context window for the same content. One analysis found a nominal 128,000-token window holding the equivalent of about 82,000 English tokens for Bengali. [18] A World Bank-affiliated team estimated that roughly 1.5 billion people, speaking languages concentrated in lower-middle-income countries, could face costs four to six times higher than English speakers — while getting worse output for it. [19]
One honest caveat, since this article is about being precise. Cost and context effects are straightforward arithmetic. The commonly-added third claim — that responses are proportionally slower — rests on much thinner evidence: the only wall-clock measurement I could find dates from 2023 and was run on an encoder model, not a modern chatbot. [15] Everything published since derives latency from the token count rather than timing it. It is probably true. It has not really been measured.
There is at least a known fix. A parity-aware version of BPE, presented at ACL 2026, changes the merge rule to favour whichever language is currently worst served, and cuts the inequality between languages by up to 89% with negligible loss of overall compression. [20] Nobody has shipped it in a frontier model.
Is tokenisation going away?
Possibly. It is the ugliest part of the stack and everyone knows it.
Three serious attempts to remove it: MambaByte works directly on raw bytes and reported competitive results against subword Transformers; [21] Meta’s Byte Latent Transformer was the first to match tokenisation-based performance at scale, tested up to 8 billion parameters; [22] and H-Net, which learns where to cut as part of training rather than deciding in advance, reported beating an equivalent BPE-based Transformer on matched compute — with the largest gains on Chinese, code and DNA, and notably better character-level robustness. [23]
So has anyone shipped one? No. As of August 2026 every frontier model you can actually use still runs on a tokeniser, and the evidence for that is the vendors’ own documentation: OpenAI still maintains and ships a BPE library, [1] Anthropic’s docs describe changing its tokeniser rather than removing it, [12] and Google confirms Gemma and Gemini share a SentencePiece tokeniser with a 262,000-entry vocabulary. [24]
The gap between “beats BPE in the lab” and “is in the product you use” is currently about two years and counting.
Tokens are fragments of text, chosen because they were statistically common, swapped for numbers before the model sees anything. They are the unit you are billed in, the unit your context window is measured in, and the reason the same sentence can cost twice as much in one language as another.
They are also the thing everyone blames for the strawberry problem — and on the current evidence, that blame is largely misplaced. Tokenisation makes character-level information awkward to reach. But the best measurement to date finds it only weakly related to counting errors, and the mechanistic work suggests the model often computes the right answer and then suppresses it on the way out.
Which is a more useful thing to know than the party trick. The failure is not that the model cannot see. It is that something in the last few layers stops it saying what it saw.
Sources & References
A note on this article. An earlier version of this page used “chunks” to mean three different things at once — tokenisation, document splitting for retrieval, and modular system design — and contained no evidence for any of it. It also implied that chunking was what solved limited context windows, which is not what happened; context length grew through architectural work. This is a complete rewrite, narrowed to one topic. Document chunking for retrieval is covered separately in our guide to building a local research assistant, which has the current benchmark numbers.