What is an embedding in AI?
Also called: Embeddings, Vector embedding, Embedding vector, Embedding (vector), AI embedding
In one minute
An embedding in AI is a list of numbers that a word, sentence, picture or other item is turned into, so that similar things end up close together; the word also means turning them into such lists.
AI works by calculating with numbers. Text-writing AIs such as ChatGPT also cut text into small pieces (tokens) and replace each one with a list of hundreds to thousands of numbers before calculating. That list is the embedding. People don’t choose the numbers; they are set as the AI learns from lots of text.
Words used in similar ways get similar lists. Words used in similar ways usually have similar meanings, so words with similar meanings gather close together. But not always: opposites such as “strong” and “weak” also end up close.
A place on a map is fixed by two numbers, across and up–down. An embedding is like a map where each place is fixed by hundreds of numbers, a map no one can see. Using this closeness, you can find text with a similar meaning even when it shares no words.
A common mix-up
“An embedding is just a number an AI gives to each word.”
Only partly right. An AI cuts text into tokens (small pieces of words) and replaces each one with a number. Up to that point, it really is just numbering. But the numbers themselves carry no link in meaning: even if “cat” and “dog” had neighboring numbers, that wouldn’t mark them as similar. An embedding is a list of hundreds of numbers assigned to each of those token numbers. During training, the numbers in each list are adjusted little by little, and words used in similar ways come to have similar lists. The token number is a name tag that tells words apart; the embedding is a set of numbers that also shows how close words are.
Explained in pictures
You’re free to use these figures in lessons, slides and posts if you credit Naru News (naru.news). CC BY 4.0.
Think of it like this
A supermarket where similar goods sit side by side
In a supermarket, similar goods are shelved near each other. Next to the milk you find yogurt and cheese, and near the apples you find oranges. So even in a store you’ve never been to, once you know where the milk is, you can guess roughly where the cheese is.
An embedding is like giving each word a spot in such a store, with its address written in numbers. Words used in similar ways are placed close together. To find text that fits a question, you just look at the shelves near the spot where the question lands.
But the comparison has limits. In a store, people decide the layout, and aisles have names such as “Dairy.” In an embedding, the layout is set by training, not by people, and each of the numbers that fix a spot usually has no name. A store has a flat floor, but an embedding’s spots lie in an invisible space defined by hundreds to thousands of numbers. And words with opposite meanings, such as “strong” and “weak,” tend to be placed close together, because they are used in similar sentences, like “Our team is strong today” and “Our team is weak today.”
Key points
Why is it called an “embedding”?
To embed something is to set it firmly inside something else. In AI, placing a word, sentence or picture as a single point inside an invisible space made of many numbers is called embedding it. That space is called the embedding space, and the list of numbers that gives the point’s location is the embedding, also called an embedding vector. A vector is a quantity with a direction and a size, and it can be written as a list of numbers. On a map, for example, an arrow that goes “3 east and 2 north” can be written as the list of two numbers 3 and 2.
How many numbers the list holds is called its number of dimensions. A place on a map is fixed by two numbers, east–west and north–south, so a map is two-dimensional. Word embeddings often use lists of 256, 512 or 1,024 numbers. People can’t draw spaces with four or more dimensions, so explanatory figures show only how close things are, on a two-dimensional map. Just as with two points on a map, how close two words are is measured by how far apart their lists of numbers are (there are several ways to measure this).
It has nothing to do with other kinds of “embedding.” Embedding a video or a map in a web page, or the computers built into appliances and cars (embedded systems), come from the same English word, embed, but they mean different things. On this page, an embedding is the list of numbers that stands for a word or picture inside an AI.
How is it different from a token?
A token is a small piece of text that an AI cuts sentences into when it reads (sometimes a whole word such as “hello,” sometimes only part of a word or character). The AI gives each token a number. The embedding is the list of numbers prepared for each of those token numbers. Inside the AI there is a large table with one row of numbers for each token number, and the AI looks up the row by the number. The numbers in the table are set by training.
In the 2017 paper in which Google researchers introduced the Transformer (the design that most of today’s text-writing AIs use), incoming tokens are turned into lists of 512 numbers with learned embeddings. A list of the same length that marks each token’s position is added to it. Adding two lists means adding the numbers in the same places: the first to the first, the second to the second, and so on. After that, the same kind of calculation is repeated many times.
At the entrance, each token has just one embedding. But as the calculation is repeated, the lists change to fit the surrounding words. The English word “orange,” for example, can mean a color or a fruit, and it ends up with a different list in each sentence. Embeddings like these, which change with the sentence, are called contextual embeddings.
How are the numbers decided?
People don’t set them one by one; training does. A table that looks up a list of numbers for each token number is placed at the AI’s entrance, and the AI is given lots of practice problems. So that it answers them correctly, the numbers in the table are adjusted little by little along with the rest of the AI.
A well-known example is word2vec, published in 2013 by Google researchers. Its practice problems use lots of text: guessing a word from the words around it, or guessing the surrounding words from the word in the middle. In the paper, it was trained on news articles with about 6 billion tokens.
With this practice, words that tend to appear surrounded by similar words end up with similar lists. In “They rode a burro down into the Grand Canyon” and “They rode a horse down into the canyon,” for example, burro and horse appear in similar sentences, so they end up close together. Nobody taught it meanings; the closeness grows out of words being used in similar ways.
Explanatory figures sometimes line words up on axes with names such as “how animal-like” or “how big.” But in real embeddings set by training, a single number rarely has such an easy-to-understand meaning. Sometimes you can guess what one stands for, but not always.
Can you really add and subtract words?
To some extent, yes. In a 2013 paper, with word lists learned in training, taking “King,” subtracting “Man” and adding “Woman” gave a list very close to that of “Queen.” The subtracting and adding are done on the numbers in the same places. The difference between male and female had been picked up automatically as a shift in a set direction. The system they trained had been taught nothing about grammar or meaning.
There may be no word exactly at the calculated spot, so the answer is the word closest to it. In another paper the same year, “Madrid − Spain + France” came closer to “Paris” than to any other word, and “Germany + capital” came close to “Berlin.”
But it doesn’t always work. In the paper with the King example, the system was given 8,000 questions about grammar relationships, such as “good is to better as rough is to what?”, and it answered close to 40% of them correctly.
What is it used for?
Besides its role inside text-writing AIs, its best-known use is searching by meaning. Ahead of time, documents are cut into short pieces, each piece is turned into a list of numbers, and the lists are stored. When a question comes in, it is turned into a list the same way, and the pieces with the closest lists are pulled out. Even if they don’t use the same words, pieces with a similar meaning can be found.
A 2020 paper used this to find passages in Wikipedia that contain the answer to a question. Wikipedia was split into about 21 million short passages, and questions and passages were each turned into lists of numbers. On most of the test sets, the share of questions with a passage containing the answer among the top 20 search results was 9 to 19 points higher than with a method that looks for exact word matches. The paper says, for example, that it can better match “bad guy” in a question with “villain” in a passage.
In the same year, the paper that named retrieval-augmented generation (RAG), where an AI looks up and reads documents before answering, also used an index of Wikipedia turned into lists of numbers in this way (more on the “Retrieval-augmented generation (RAG)” page).
Other uses include grouping similar things, picking recommendations, spotting things that are far from all the others, and sorting things into set categories. OpenAI’s AIs that make embeddings of text usually return a list of 1,536 or 3,072 numbers (the lists can also be made shorter).
What are its weak points?
Closeness is learned from how words are used, so it doesn’t always match the meaning people feel. For example, “strong” and “weak” have opposite meanings, but they are used in similar sentences, such as “Our team is strong today” and “Our team is weak today.” So embeddings learned from usage often fail to tell words with opposite meanings apart.
In embeddings that give each word only one list, a word with several meanings can only be shown as a single point. Contextual embeddings, whose lists change with the sentence, can show the same word with a different list in each sentence.
Leanings in the training text are copied too. A 2016 paper examined an embedding trained on Google News articles. Asked to complete “man is to computer programmer as woman is to x,” it answered “homemaker.” The paper warns that because such embeddings are so widely used, they tend not just to copy the gender stereotypes in text but to spread them further.
Search has weak points too. When an exact word matters, such as a rare name or a code number, searching for exact word matches can do better. So the two ways are sometimes combined. In the 2020 paper that searched Wikipedia for passages containing answers, combining them improved the results on some tasks.
Check yourself
Q1An AI embedding places a word as a single point in a map-like space. Yet this space can’t be drawn the way an ordinary map can. Why?
Answer and explanation
Answer: B Each point’s place is set by hundreds of numbers
In an embedding, each word’s place is set by a list of hundreds to thousands of numbers. A place on a map is set by two numbers and a place in a room by three, but people can’t draw a space where places are set by four or more numbers. So explanatory figures use just two numbers and show only how close things are on a flat map. Lots of points would not stop you from drawing a flat map, and the places are set by learned numbers, not by the shapes of the letters.
Q2Retrieval-augmented generation (RAG) is a system where an AI looks up and reads documents before answering. When it searches with embeddings, how does it choose the documents that fit a question?
Answer and explanation
Answer: C It turns the question into an embedding too and picks the closest
The documents are each turned into an embedding (a list of numbers) ahead of time and stored. When a question comes in, it is turned into a list the same way, and the documents with the closest lists are pulled out and given to the AI along with the question. Because the search goes by closeness in meaning, documents that don’t use the same words can still be found. The stored embeddings aren’t adjusted for each question. Picking the documents where the question’s words appear most is a different method that matches words directly, not one that uses embeddings.
Q3“Strong” and “weak” have opposite meanings. In word embeddings learned from lots of text, where do these two words tend to end up?
Answer and explanation
Answer: A Close together, because they appear in similar sentences
Words with opposite meanings are used in similar sentences, such as “Our team is strong today” and “Our team is weak today.” Embeddings learn from how words are used, so words used in similar ways end up close together, and opposites tend to end up close too. So such embeddings often fail to tell words with opposite meanings apart. Closeness in an embedding is not always the same as the closeness in meaning people feel.
Related terms
- Token
- Large language model
- Retrieval-augmented generation (RAG)
- Transformer
- Neural network
- Machine learning
- Deep learning
- Parameter
- Neuralese
- Persona vectors
- Vector
- Latent space
- Vector database
- Word2vec
Sources and how we checked
- Mikolov, Yih and Zweig, "Linguistic Regularities in Continuous Space Word Representations" (Microsoft Research; NAACL-HLT 2013, pp. 746–751, June 2013; abstract, Sections 1, 3, 4.1, 5 and 6; read November 12, 2026)
- Mikolov et al., "Efficient Estimation of Word Representations in Vector Space" (Google; arXiv, January 2013; Section 1.1 and Sections 3 and 4; read November 12, 2026)
- Mikolov et al., "Distributed Representations of Words and Phrases and their Compositionality" (Google; arXiv, October 2013; abstract and Sections 1, 2 and 4; read November 12, 2026)
- Vaswani et al., "Attention Is All You Need" (Google Brain, Google Research and others; arXiv, June 2017; Sections 3.1, 3.4 and 3.5 and Table 3; read November 12, 2026)
- Google, "Embeddings" (Machine Learning Crash Course; the trouble with one-hot encoding and what an embedding is; read November 12, 2026)
- Google, "Embeddings: Embedding space and static embeddings" (Machine Learning Crash Course; words used in similar contexts sit closer, typical sizes of word embeddings (256, 512, 1,024) and what single dimensions mean; read November 12, 2026)
- Google, "Embeddings: Obtaining embeddings" (Machine Learning Crash Course; embedding layers, the limit of one embedding per word, and contextual embeddings; read November 12, 2026)
- Google, "Machine Learning Glossary" (entries for embedding layer, embedding space and embedding vector; read November 12, 2026)
- OpenAI, "Vector embeddings" (API documentation; uses of embeddings and the vector lengths of text-embedding-3-small and text-embedding-3-large; read November 12, 2026)
- Karpukhin et al., "Dense Passage Retrieval for Open-Domain Question Answering" (Facebook AI, University of Washington and Princeton University; EMNLP 2020; arXiv, April 2020; abstract, Sections 1, 3.1, 4.1 and 5 and Table 2; read November 12, 2026)
- Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks" (Facebook AI Research, University College London and New York University; NeurIPS 2020; arXiv, May 2020; abstract, Section 2.2 and Section 3; read November 12, 2026)
- Ono, Miwa and Sasaki, "Word Embedding-based Antonym Detection using Thesauri and Distributional Information" (Toyota Technological Institute; NAACL-HLT 2015, pp. 984–989; abstract and Section 1; read November 12, 2026)
- Bolukbasi et al., "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings" (Boston University and Microsoft Research; NIPS 2016; arXiv, July 2016; abstract and Section 1; read November 12, 2026)
- IBM, Japanese page “What is embedding?” (for the Japanese names “umekomi,” “embedingu” and “umekomi bekutoru”; read November 12, 2026)
- AWS, Japanese page “What are embeddings in machine learning?” (for the Japanese names “umekomi” and “umekomi bekutoru”; read November 12, 2026)
AI wrote this page from the sources above, and a different AI checked it against the text of those sources (November 12, 2026). A person with a master’s degree in science and a Japanese high-school science teaching license also read and checked it before publishing. Numbers are as of that date. If anything was hard to follow, you spot a mistake, or there’s a term you’d like explained, please write to [email protected].
AI terms
- AI agent
- AI alignment
- Alignment faking
- Artificial general intelligence (AGI)
- Artificial intelligence (AI)
- Autoformalization
- Backdoor
- Benchmark
- Chain of thought (CoT)
- Context window
- Data poisoning
- Deep learning
- Deepfake
- Diffusion model
- You are hereEmbedding
- Emergent misalignment
- Evaluation awareness
- Fine-tuning
- Formal verification
- Generalization
- Generative adversarial network (GAN)
- Generative AI
- Gradual disempowerment
- Hallucination
- Hyperparameter
- Intelligence explosion
- Interpretability
- Jailbreaking
- Large language model (LLM)
- Latent space
- Lean
- Machine learning
- Mixture of experts (MoE)
- Neural network
- Neuralese
- Overfitting
- Parameter
- Persona vectors
- Pre-training
- Prompt
- Prompt injection
- Reasoning model
- Recursive language model (RLM)
- Recursive self-improvement (RSI)
- Red teaming
- Reinforcement learning (RL)
- Retrieval-augmented generation (RAG)
- Reward hacking
- Shutdown resistance
- Sleeper agent
- Subliminal learning
- Superintelligence (ASI)
- System prompt
- Technological singularity
- Time horizon
- Token
- Transfer learning
- Transformer