In the English sentence “The animal didn’t cross the street because it was too tired,” a Transformer can tell that “it” means the animal thanks to a calculation called attention. What does attention do?

Compares how strongly other words bear on it here

Compares how far away other words sit

Compares how alike other words are in the dictionary

In the 2017 paper, the Transformer adds a mark to each word’s list of numbers before sending it through the layers. What information does that mark carry?

Where the word comes in the sentence

Which words the word is often used with

What role the word plays, such as the subject

Why could the Transformer in the 2017 paper train faster than the older design (the RNN)?

It can run the words’ calculations at the same time

It splits words into single letters before calculating

It uses just one layer instead of a stack

Home › Glossary › AI

What is a Transformer in AI?

Also called: Transformer model

Term explainer · Checked November 8, 2026 · 11 min read

In one minute

It is a way of building AI (a design for neural networks) that works out how strongly the words in a text are related to each other by comparing all of the words at once.

Google researchers introduced it in 2017 in a paper called “Attention Is All You Need.” In “The animal didn’t cross the street because it was too tired,” you can only tell that “it” means the animal by looking at the other words. This calculation, deciding which words to look at and how much, is called “attention.”

The older design (the RNN) read a sentence one word at a time, so the calculations within a sentence could not run side by side. A Transformer can run them at the same time, so it trains faster. It is now the most widely used design for large language models (LLMs), and the T in ChatGPT’s GPT stands for Transformer.

A common mix-up

“A Transformer reads a sentence one word at a time, from front to back.”

Half right. When a model like ChatGPT writes, it does produce one token (a chunk of text the AI counts as one unit) at a time, adding what it has written to its input before producing the next. But reading works differently. The older design (the RNN) read a sentence from left to right, one word at a time, and could not move on to the next word until the calculation for the one before was done. In 2019, Google described Transformers as “models that process words in relation to all the other words in a sentence, rather than one-by-one in order.” Because the relationships between words can be calculated at the same time, training goes faster.

Explained in pictures

A figure showing attention (the example sentences are from Google’s 2017 post; the line widths are made up for this page). Title: “Which word does ‘it’ look at?” At the top, the sentence “The animal didn’t cross the street because it was too ___.” and “The last word decides what ‘it’ means.” Left box, “… too tired.”: from “it” at the bottom, a thick solid line labeled “Looks a lot” goes up to “animal,” and a thin dotted line labeled “A little” goes to “street.” Below: “it = the animal.” Right box, “… too wide.”: a thick solid line labeled “Looks a lot” goes from “it” to “street,” and a thin dotted line labeled “A little” goes to “animal.” Below: “it = the street.” Band at the bottom: Attention works out, for each sentence, how much to look at each word. — What “it” refers to depends on the last word. If it was “too tired,” it is the animal; if “too wide,” the street. A Transformer compares, in numbers, how strongly “it” is related to the other words and looks hardest at the most related one (thick solid line). In its 2017 post, Google showed a trained model’s attention from “it” going mostly to “animal” in one sentence and to “street” in the other. In French, “it” becomes il or elle depending on which noun it means, so this is needed to translate correctly (line widths are made up for this page).

A figure comparing reading one word at a time with comparing all words at once (the example sentence is made up for this page). Title: “One word at a time, or all at once?” Left, “Older way (RNN)”: four words, “The,” “dog,” “ran” and “home,” joined in a row by arrows and numbered ① to ④ above. Below the arrows: “→ passes its result to the next word.” Then “Each word waits for the one before” and “‘The’ and ‘home’: 3 steps apart.” Right, “Transformer”: the same four words, with curved lines joining every pair (six in all), labeled “All links computed at once.” Below: “→ any two words are 1 link apart,” “Compares all the words at once” and “‘The’ and ‘home’: 1 step apart.” Band at the bottom: Computing at once makes training faster (2017 paper). — The older design on the left (the RNN) reads one word at a time and passes its result on to the next word. Each word has to wait for the one before, and linking words far apart takes many steps. The Transformer on the right compares every pair of words at once (curved lines). Any two words are linked in one step, and the calculations can run at the same time, so training got faster (the sentence is made up for this page).

The words going in are turned into lists of numbers, marked with their positions and passed through many layers of the same shape. In each layer, ① attention takes in information from the other words, and ② the same calculation is done for each word on its own. At the end, the model gives a probability for each possible next word, picks one, adds it to the end and starts again. That is how it writes, one token at a time. The 2017 translation model had 6 layers (6 for reading and 6 for writing), and GPT-3 (2020) had 96 (the words and percentages are made up for this page).

Think of it like this

A game of telephone vs. a meeting where everyone can see everyone

The older design (the RNN) is like a game of telephone. Starting from the first person in line, each person sums up what they heard and passes it to the next. Nobody can start until the person before them is done. In a long line, what was said near the start gets harder to pass along as it goes through more and more people.

A Transformer is more like a meeting where everyone looks at the same sheet of paper and talks at once. Each person checks, all at the same time, whose words matter most to them, and can listen directly to someone sitting far away. Because everyone works at the same time, it finishes sooner.

The comparison has limits. A Transformer’s “attention” is a number it calculates, not the same as a person paying attention. And if the number of people at the meeting doubles, the number of pairs roughly quadruples. That is why longer texts take much more calculation.

Key points

What does attention actually calculate?

Google’s machine learning course explains that attention within a single text (self-attention) asks, for each word, “How much does each other word affect the interpretation of this word?” The answer comes out as numbers, bigger for more closely related words.

Inside an AI, each word is represented as a long list of numbers. In the 2017 paper, three more lists of numbers are made from each word’s list: a query (like what it is looking for), a key (like a label) and a value (like the content). The query of one word is compared with the keys of all the words; the better the match, the bigger the weight, and the values are mixed together using those weights. The result is a new list of numbers that has taken in information from the surrounding words.

In its 2017 post, Google gave the example “I arrived at the bank after crossing the river.” “Bank” can mean a place for money or the side of a river. A Transformer can learn to attend right away to “river,” Google explained, and so decide in a single step that it means the riverbank.

This comparison is done several ways side by side (“multi-head” attention). The 2017 paper’s base model used eight, and its big model sixteen. The paper’s appendix shows some that seem related to the structure of the sentence, and some that seem to work out what words like “its” refer to.

How is it different from the older design (the RNN)?

Around 2017, the main models for jobs that turn one sequence into another, like translation, were built on RNNs (recurrent neural networks) or convolutional neural networks, and the best of them also used attention. The paper’s title, “Attention Is All You Need,” fits what it proposed: a design built only on attention, with no RNN and no convolution. It was written by eight people: researchers at Google Brain and Google Research, and one from the University of Toronto.

An RNN reads a sentence from left to right (or right to left), one word at a time, passing the results so far on to the next word. The paper says this step-by-step nature keeps the calculations within a sentence from running in parallel. Linking words that are far apart also takes many steps, and Google’s post says the more steps that takes, the harder it is for an RNN to learn. In a Transformer, any two words are linked directly in one round of attention.

In the paper, English-to-German translation scored 28.4 (BLEU, a translation score where higher is better), beating the previous best by more than 2 points. English-to-French scored 41.8 after 3.5 days of training on eight GPUs (computer chips for calculation). The paper says that even about 12 hours of training on eight GPUs was enough to reach a new best in translation quality. Google said training was up to about ten times faster (an order of magnitude; the company’s own claim).

How does it know the order of the words?

Attention just compares words with each other, so on its own it has no sense of which word comes where. But “the dog chased the cat” and “the cat chased the dog” use the same words and mean opposite things (an example made up for this page).

So the 2017 paper, noting that the model has no RNN or convolution and must be given information about position to use word order, added a mark for each word’s position (“positional encoding”) to that word’s list of numbers. The marks are made with wave-shaped functions (sine and cosine, the functions that describe waves).

The paper says it also tried marks that the model learns during training and got nearly identical results. It chose the wave-shaped version because it may let the model handle texts longer than any it saw in training.

What does it look like inside?

The words going in (tokens) are first turned into lists of numbers. Position marks are added, and the result passes through many layers of the same shape. Each layer has two parts: one that takes in information from the other words using attention, and one that does the same calculation for each word separately.

The 2017 translation model had 6 layers on the reading side (the encoder) and 6 on the writing side (the decoder). Each layer on the writing side also has one more attention part, which looks at what the reading side produced. According to Google’s course, the encoder turns the input text into an in-between representation, and the decoder turns that into useful text. The paper’s models had about 65 million parameters (the numbers used in the calculation) in the base version and about 213 million in the big one.

Later, designs that use only one side also appeared. BERT, which Google introduced and released in 2018, stands for “Bidirectional Encoder Representations from Transformers.” It reads a text by looking at the words both before and after each word. Others use only the decoder, producing new tokens from the text already written (Google’s course).

How does it write text?

One token at a time. The 2017 paper says the model is “auto-regressive”: when producing the next piece, it uses what it has produced so far as extra input. Google’s post also says the decoder produces one word at a time, from left to right. The output of the last layer is turned into a probability for each possible next word.

On the writing side, the attention is masked so that each word cannot look at the words after it (the paper’s “masking”). The paper says this makes the prediction for each position depend only on the known words before it. Google’s course says apps that generate text token by token need this kind of one-way attention. The reading side (the encoder) looks both before and after.

Google’s course also describes large language models (LLMs) as autocomplete mechanisms that can predict thousands of tokens. According to OpenAI’s paper, GPT-3 (2020) was an “autoregressive language model” with 175 billion parameters, 96 layers and room to read 2,048 tokens at once. GPT stands for “Generative Pre-trained Transformer” (a 2023 paper by researchers at OpenAI and elsewhere).

Is it used for things other than words?

Yes, for photos. In 2020, Google researchers cut photos into small squares (patches), such as 16 × 16 pixels, lined them up like words (tokens) and fed them to a Transformer (the Vision Transformer, or ViT). The paper says the design followed the original Transformer as closely as possible. After pre-training on large image collections, it got excellent results compared with the best convolutional neural networks, which had long been used to recognize photos, while needing less computing to train.

And for voices. Whisper, a speech recognition system OpenAI announced in 2022, is a Transformer with a reading side and a writing side. It cuts audio into 30-second pieces and turns them into a picture of how strong each pitch is over time (a spectrogram) before feeding them in. It was trained on 680,000 hours of audio with transcripts.

The 2017 paper itself ended by saying the authors planned to extend the Transformer beyond text, to large inputs and outputs such as images, audio and video. By 2020, the ViT paper said the Transformer had become the “de-facto standard” for language tasks.

Does it have weak spots?

The longer the text, the faster the calculation grows. Attention compares every pair of words, so when the number of words doubles, the number of pairs roughly quadruples, and ten times the words means about a hundred times the pairs. Google’s course likewise says that, with N tokens in the context, the work grows with N², and the numbers of layers and heads multiply on top of that.

For very long texts, the 2017 paper suggested limiting each word to looking at a set number of nearby words. The trade-off is that it then takes more steps to link words that are far apart.

Check yourself

Q1In the English sentence “The animal didn’t cross the street because it was too tired,” a Transformer can tell that “it” means the animal thanks to a calculation called attention. What does attention do?
Answer and explanation
Answer: B Compares how strongly other words bear on it here
For each word, attention compares in numbers how much the other words affect how that word is read. In the “too tired” sentence, “it” looks hardest at the animal; in the “too wide” sentence, at the street. Going by distance would pick “street,” right before “it,” and how alike words are in the dictionary stays the same whatever the sentence says.
More on this → Key points: What does attention actually calculate?
Q2In the 2017 paper, the Transformer adds a mark to each word’s list of numbers before sending it through the layers. What information does that mark carry?
Answer and explanation
Answer: C Where the word comes in the sentence
Attention alone does not carry word order. Because order changes meaning, as in “the dog chased the cat” and “the cat chased the dog,” a mark showing each word’s position (positional encoding) is added.
More on this → Key points: How does it know the order of the words?
Q3Why could the Transformer in the 2017 paper train faster than the older design (the RNN)?
Answer and explanation
Answer: A It can run the words’ calculations at the same time
An RNN reads a sentence one word at a time and cannot move on until the word before is done, so the calculations within a sentence could not run in parallel. A Transformer can calculate the relationships between words at the same time. In the paper, even about 12 hours of training on eight GPUs reached a new best in translation.
More on this → Key points: How is it different from the older design (the RNN)?

Sources and how we checked

AI wrote this page from the sources above, and a different AI checked it against the text of those sources (November 8, 2026). A person with a master’s degree in science and a Japanese high-school science teaching license also read and checked it before publishing. Numbers are as of that date. If anything was hard to follow, you spot a mistake, or there’s a term you’d like explained, please write to [email protected].