What is a mixture of experts (MoE)?

Google’s machine learning glossary describes a mixture of experts as “a scheme to increase neural network efficiency by using only a subset of its parameters (known as an expert) to process a given input token or example.” So an “expert” is not a person or a separate AI, but a group of the numbers an AI calculates with (parameters). Mistral AI, which released Mixtral 8x7B, likewise describes its 8 experts as “8 distinct groups of parameters.”

In Mixtral, the model is a stack of 32 layers, and one part of each layer (a block called the feedforward block) is split into 8 experts. When a token arrives, the router (also called a gate) gives each of the 8 a score and picks the 2 with the highest scores. Their results are added together with weights set from the scores and passed to the next layer. The router, too, is a part whose numbers are set during learning.

Because this happens at every layer and for every token, even within one passage a different token may call different experts. The paper puts it this way: “Even though each token only sees two experts, the selected experts can be different at each timestep.”

Multiplying the name “8x7B” gives 56 billion, but the paper’s figure is 47 billion in all (46.7 billion in the company’s announcement). That is because only one part of each layer is split into experts; the other parts are not.

Because we want bigger AIs (more parameters) without more calculation. A 2017 paper by Google Brain researchers opens with: “The capacity of a neural network to absorb information is limited by its number of parameters.”

But an ordinary AI uses all of its parameters for every input. The more numbers you add, the more calculation you need. The fix is to run only part of the network for each input (“conditional computation,” in the paper’s words). The paper reports that an MoE with thousands of experts gave more than a 1000x improvement in model capacity with only minor losses in computational efficiency.

The Mixtral paper likewise says the technique “increases the number of parameters of a model while controlling cost and latency.” Using one fifth as many parameters per token as Meta’s Llama 2 70B, Mixtral outperformed Llama 2 70B across most categories of the tests in the paper.

The name makes it sound as if there is a math expert or a Japanese expert. But nobody assigns the jobs; they split up during learning. So what did they split by? Two studies looked.

The Mixtral paper checked whether experts had specialized in particular fields during training. For science papers (arXiv), biology and medical abstracts (PubMed) and philosophy texts (PhilPapers), the share of times each expert was picked was very similar, and the paper says: “Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic.” (Only a set of math problems, DM Mathematics, was slightly different.)

Instead, the several tokens that make up one word, such as “self” in Python or “Question” in English, often went to the same expert. In the examples shown, especially at the first and last layers, indentation tokens in code always went to the same experts. Tokens in a row were also often handled by the same expert. The paper says the choice of experts seems more aligned with the syntax (the form of sentences) than with the field.

In Google researchers’ ST-MoE (2022), experts on the side that reads text were found with jobs like punctuation only, verbs only, mostly numbers, or mostly pieces of names (proper nouns). On the side that writes text, such jobs were far less noticeable. In a model trained on many languages, the Japanese “de” and “nanode” went to the same expert as the English “and,” and the paper says it found no evidence of language specialization.

These are findings from the two models studied, though. In 2024, DeepSeek researchers proposed a design (DeepSeekMoE) that splits experts into finer pieces and sets aside “shared experts” for common knowledge, so that each expert can have a clearer job of its own.

There is a trade-off. Experts that are not called may be needed for the next token, so all of them must be kept in memory (like a desk you work on). The Mixtral paper says the memory needed to run it is proportional to all 47 billion parameters, not the 13 billion used per token (though that is still smaller than Llama 2 70B).

Speed may also not rise as much as the reduced calculation suggests. The paper warns that the routing mechanism adds extra overhead, and that MoE models are more suitable for jobs that process many texts together in batches.

Memory matters when you run a big MoE on your own machine, too. OpenAI’s model card for gpt-oss-120b (116.8 billion parameters, 5.1 billion used per token) says that by storing the numbers in a smaller form (quantization), the model fits on a single GPU (a calculating chip) with 80GB of memory.

Left alone, it does. The 2017 paper says the router tends to end up always giving large weights to the same few experts, and that “this imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more.”

So the researchers added an extra term to the training loss (the penalty the AI learns to reduce) that encourages all experts to have equal importance. Switch Transformers (Google, 2021) also adds an auxiliary loss to encourage a balanced load across experts. In that study, if more tokens are sent to an expert than its set limit, the overflow tokens skip that expert’s calculation and are passed straight to the next layer.

The DeepSeek-V3 technical report (2024) says that too large an auxiliary loss will impair model performance, and uses another way. Each expert gets a bonus (a bias) added to its router score; at the end of each training step, the bias of an overloaded expert is lowered and that of an underloaded one is raised. The bias is used only to choose which experts to call.

The idea is old. It goes back to the 1991 paper “Adaptive Mixtures of Local Experts” by Jacobs, Jordan, Nowlan and Hinton (MIT and the University of Toronto), which describes systems of many separate networks, “each of which learns to handle a subset of the complete set of training cases.” On a vowel-recognition task, they write, learning divided the work into suitable subtasks, each solved by “a very simple expert network.”

In 2017, Shazeer and colleagues at Google Brain used this in large language AIs. Their “sparse” MoE lets a router pick only a few of up to thousands of experts, with up to 137 billion parameters. They tested it on predicting the next words in text and on translation.

Switch Transformers (Google, 2021) made routing simpler by sending each token to only one expert, and reports up to 7x faster pre-training on the same computing resources. Its largest model had 2048 experts and 1.6 trillion parameters. Later, Mistral AI released Mixtral 8x7B in December 2023 and published the paper describing it in January 2024.

These are some that their makers describe as MoE in papers or docs: Mixtral 8x7B (Mistral AI; 2 of 8 per layer). DeepSeek-V3 (DeepSeek, December 2024; of 671 billion parameters, 37 billion used per token; in its expert layers, 8 of 256 experts plus 1 shared expert). gpt-oss-120b and gpt-oss-20b (OpenAI, August 2025; for each token, 4 experts are picked from 128 in the 120b and from 32 in the 20b). The gpt-oss models are open-weight: their parameters are public.

Google’s Gemini 1.5 technical report (2024) calls Gemini 1.5 Pro “a sparse mixture-of-expert (MoE) Transformer-based model,” and the Gemini 2.5 technical report (2025) also says the Gemini 2.5 models are sparse MoE. Neither gives the number of experts or parameters.

In its GPT-4 technical report (2023), OpenAI says that, given the competitive landscape and safety implications, the report contains no further details about the architecture (including model size). For Claude (Anthropic), too, we did not find an official document saying whether it is an MoE when we made this page. So we can’t say here whether ChatGPT or Claude is an MoE inside.