What is a token in AI?

If you keep a list of whole words, you get stuck when a new word or an unusual name shows up that is not on the list. If you cut everything into single characters instead, you can read any word, but the text turns into a very long line of pieces that takes more work to handle.

So common runs of letters become one big piece, and rare words become a combination of small pieces. The AI company Anthropic also explains that bigger pieces are more efficient, while smaller pieces let a model handle words it has never seen. For example, “tokenization” is cut into “token” and “ization” (GPT-4o’s tokenizer).

No one picks them one by one. They are made automatically from lots of text. In a common method, you start with the smallest units, each as its own piece. Then you find the two pieces that sit next to each other most often in the text and join them into one new piece. You repeat this until the set reaches a chosen size.

For example, if “in” is followed by “g” very often in lots of text, the two are joined into one piece, “ing.” OpenAI’s guide to its tokenizer, too, says “ing” is a common piece in English, so “encoding” is cut into “encod” and “ing.” That way, the more often a run of letters shows up, the more likely it is to become one big piece. The same idea also works for text in languages like Japanese, which do not put spaces between words.

In Japanese, too, the number of characters and the number of tokens do not line up. GPT-4o’s tokenizer turned the nine characters of 私は猫が好きです。 (“I like cats.”) into six tokens: 私は, 猫, が, 好き, です and 。. The four-character Japanese word トークン (“token”) became three tokens: ト, ーク and ン.

The same meaning can take a different number of tokens in a different language. “I like cats.” was four tokens, and 私は猫が好きです。, which means the same, was six (both with GPT-4o’s tokenizer). With the older GPT-4 tokenizer, the same Japanese sentence was ten tokens, and the single character 猫 (“cat”) was split into three (the figure caption explains why).

One thing is the limit on how much an AI can read and write at once. This is called the “context window,” and it is like the AI’s “working memory” (Anthropic’s explanation). For example, Claude’s newer models can handle 1 million tokens (November 2026). Google’s guide describes 1 million tokens as, among other things, “8 average length English novels” (that is for English; other languages give different token counts). A very long document may not fit.

The other is price. Where you pay for how much you use (for example, when a company builds an AI into its own app), the price is set by the number of tokens, and the tokens the AI reads and writes are counted separately. With Claude, the tokens it writes cost five times as much as the tokens it reads (November 2026).

Because usually, each time the AI writes a reply, it reads the whole conversation from the start (Anthropic’s explanation). In a chat with 10 back-and-forths, the 10th reply also reads the 9 earlier ones. Some apps drop or shorten the oldest parts to keep the reading down.

So in a long chat, the AI not only gets closer to the limit; Anthropic also writes that as the tokens pile up, it can get harder for the AI to recall what was said early on. Starting a new chat when the topic changes cuts down the reading.

No. The English word “token” originally meant “a sign” (Old English tācn), and it can also mean a coin used in place of money. Different fields use it for different things.

In crypto, a token is a digital marker whose owner can be recorded. In logging in, an access token is a string of characters that shows an app is allowed to do certain things for a set time. When people talk about AI, a token usually means the “piece of text” on this page.