What is a context window in AI?

According to Anthropic’s documentation, everything sent for one reply goes into the window: the instructions for the AI (the system prompt), every message in the conversation so far, the results of any tools it used (such as a search), images and documents, and the descriptions of the tools it can use. The answer the AI writes that turn goes in too. The window is the room for reading and writing together.

So if you give a model with a 100,000-token window a 99,000-token document, only about 1,000 tokens are left for the answer (an example made up for this page). Anthropic says that on newer Claude models (Claude 4.5 and later), if the answer reaches the window’s limit while it is being written, the model stops there. If the input alone is over the limit, the request fails with an error from the start.

For an AI that thinks before answering (a reasoning model), the thinking it does before the answer (reasoning tokens) also goes into that turn’s window and is counted as writing. Whether the thinking from one reply stays in the window for the next reply depends on the model; most newer Claude models keep it and count it like any other input (Anthropic’s explanation, November 2026).

Using a “cache,” which keeps what was already read so it can be reread quickly, still counts toward the window. What caching changes is the price and speed of reading those tokens, not whether they count (Anthropic’s explanation).

According to OpenAI’s 2020 paper, GPT-3’s window was 2,048 tokens. Google’s documentation says earlier text-generating AIs could only handle 8,000 tokens at a time, and newer ones pushed that to 32,000 or even 128,000 tokens.

In the companies’ documentation as of November 2026, Anthropic’s newer Claude models (such as Claude Opus 5.5) and Google’s Gemini 4 Argon have 1 million tokens, and OpenAI’s GPT-6 Astra has 1.05 million. In ordinary one-at-a-time use, Claude can write up to 128,000 tokens of that in a single reply.

How much is 1 million tokens? Google’s documentation compares it to “8 average length English novels,” “50,000 lines of code (with the standard 80 characters per line)” and “transcripts of over 200 average length podcast episodes,” among other things. Anthropic says that with Claude’s current tokenizer, it is roughly 555,000 words, or 2.5 million characters. But how many characters one token stands for depends on the model and the language (Anthropic’s glossary).

It depends on how the AI is used. Through Anthropic’s API (the gateway apps use to reach the AI), if the input alone is over the window, the request gets a “prompt is too long” error and no reply is made.

Chat screens may drop the oldest parts first. Anthropic writes that chat interfaces such as claude.ai can also manage the window on a “first in, first out” basis. Whatever is dropped can no longer be seen by the AI.

Another way is to summarize. Anthropic’s “compaction” replaces older turns with a summary the AI writes, so a conversation can keep going past the window’s limit. In a September 2025 article about Claude Code, an AI coding tool, Anthropic explained that a conversation nearing the limit is summarized by the AI, and the work continues from that summary plus the five most recently accessed files. But details that drop out of the summary cannot be seen later, even if they are needed. For when the summary leaves out something important, the documentation also lets you write your own instructions for how to summarize.

Having room for more is handy, but Anthropic writes that more context isn’t automatically better: as the token count grows, accuracy and recall degrade. It calls this “context rot,” and describes it as a gradual decline rather than a point where the AI suddenly cannot read (“a performance gradient rather than a hard cliff”).

In the 2023 paper “Lost in the Middle,” researchers at Stanford University and elsewhere gave AIs a question and 20 documents and moved the one document containing the answer to different places. GPT-3.5-Turbo answered correctly 75.8% of the time when that document came first and 63.2% when it came last, but only 53.8% when it was 10th, lower than the 56.1% it got with no documents at all, using only what it learned in training. With all 20 documents, the input averaged about 3,000 tokens (at most 3,920), within its 4,000-token window. The paper also says that models with extended windows are not necessarily better at using what is in them.

Google’s documentation likewise says that while a model can find a single piece of information about 99% of the time, it does not perform with the same accuracy when looking for several, and performance can vary widely. The other issue is price. Where you pay for what you use, you pay for the input tokens every time you send a long document. With OpenAI’s GPT-6 Astra, a request with more than 272,000 input tokens costs twice as much for input and 1.5 times as much for output (November 2026).

When it writes an answer, the design most of today’s AIs use (the Transformer) compares the tokens in the window with each other and works out which ones are most closely related (a calculation called attention). Double the tokens and the pairs to compare roughly quadruple; ten times the tokens, about a hundred times the pairs (just as doubling the size of a class roughly quadruples the number of possible pairs of students). The 2017 paper also shows this work growing with the square of the text’s length.

Anthropic writes that as the context gets longer, there are so many pairs that a model’s ability to capture each relationship gets “stretched thin.” It also explains that models learn how to spread their attention from their training data, where shorter texts are more common, so they have less experience with relationships that span a whole long text. A bigger window means heavier calculation, and even when the window is bigger, the model does not necessarily make good use of everything in it.

Anthropic’s documentation recommends that for long documents (about 20,000 tokens or more), you put the documents near the top of your prompt (the text you give the AI) and your question and instructions at the end. It says that in tests, putting the question at the end improved answer quality by up to 30% (the company’s own claim). Google’s documentation also says that when the context is long, the model usually does better if the question comes at the end.

Anthropic also suggests wrapping each document in its own tags (XML tags) with its source when there are several, and asking the AI to quote the relevant parts of the documents before it answers. Anthropic’s article for people building AI agents recommends finding the smallest set of useful information that gets you the result you want. However big the window, choosing what goes in matters.

It is separate from what the AI learned. Anthropic’s glossary describes the context window as the AI’s “working memory,” different from the huge body of text it was trained on. The AI cannot open the text it was trained on and look back at it while replying. What it can look back at, right then, is only the text in the window.

Anthropic’s memory tool lets the AI write down what it wants to keep in files outside the window and read them back in a later conversation when needed, so it can build up knowledge without keeping everything in the window. “Retrieval-augmented generation” (RAG), which looks up documents and puts them into the window, likewise brings in information from outside the window, only as much as is needed. Either way, in the end the AI reads it inside the window.