Mixture of experts, often shortened to MoE, is an AI design where a model contains several smaller specialist networks, and a small routing network sends each input to only one or two of them. Instead of waking up the whole model for every question, the model wakes up just the specialists most likely to help. That is why very large models built this way can feel fast, even though their total size is huge.
Key Takeaways
- A mixture of experts model holds many specialist networks, called experts, plus a router that chooses among them.
- For each input, usually only one or two experts run, so the active computing cost stays much lower than the model’s total size suggests.
- Experts tend to specialise during training, for example on certain kinds of language, code, or reasoning.
- The trade-off is memory and complexity: all experts still have to be stored, and routing has to stay balanced.
- MoE is a design choice inside a model, not a separate kind of AI on its own.
What does mixture of experts actually mean?
Think of a hospital reception desk. You do not see every doctor for a sore knee; the receptionist sends you to the specialist who handles knees. A mixture of experts model works in a similar spirit, though the comparison only goes so far. Each expert is a small neural network, a set of learned weights that is good at handling a certain slice of the input. The router, sometimes called a gating network, looks at the input and scores each expert. It then picks the top one or two and passes the work on.
In modern language models, this idea is usually applied inside the model, layer by layer. A standard transformer layer has a feed-forward part that processes every token. In a MoE version, that part is replaced by a set of experts and a router. Shared parts of the model, such as attention, still run for every token. Only the expert part is selective, which is why people describe these models as sparse: most of the model is quiet at any given moment.
How does the router choose an expert?
The router is deliberately small. For each token, or chunk of input, it produces a score for every expert, often with a softmax function that turns scores into something like probabilities. The model then keeps the highest scores, a method known as top-k routing. If k is one, only the best expert runs. If k is two, the two best run and their outputs are combined, weighted by their scores.
That sounds tidy, but training a router is fiddly in practice. Early on, the router has no good reason to prefer one expert over another, so it may send almost everything to the same few experts. Those experts then get more training, get better, and get chosen even more, while other experts sit idle and never improve. Engineers call this collapse, and they fight it with a balancing loss, an extra training signal that nudges the router to spread work more evenly.
There is also a capacity question. An expert can only handle so many tokens at once in a batch. If too many tokens are routed to the same expert, some may overflow and skip that layer altogether, travelling on through a shortcut connection instead. Setting capacity too high wastes computing power; setting it too low drops work. Getting that balance right is one of the practical skills in building MoE models.
Why would anyone build a model this way?
The main reason is scale without a matching jump in running cost. Suppose you want a model with far more knowledge and capacity than a single dense network can hold at a reasonable cost. You could simply make the dense network bigger, but then every input pays the full cost of that bigger network. With MoE, you can add experts, growing the model’s total capacity, while the cost per input grows much more slowly, because only a few experts run.
For a reader, the practical point is simpler. When a chatbot or assistant answers quickly despite being described as huge, MoE is one design that can explain that. It is not magic, and it does not mean the model is smaller than advertised. It means the model is organised so that only part of it works on your question.
What do the experts actually learn?
Experts do not arrive with job titles. Nobody labels one expert “the maths expert” and another “the French expert” before training, at least not in the usual setup. Specialisation emerges because the router keeps sending similar inputs to the same expert, and that expert keeps learning from them.
In language models, researchers have found patterns that make intuitive sense. Some experts may handle punctuation or common words, others may handle code-like text, and others may handle particular languages or topics. But the boundaries are rarely clean. An expert might be strong on a mix of things that happen to appear together in training data, rather than on one neat school subject. So it is best to think of experts as learned specialities, not as hand-written departments.
Where is mixture of experts used?
MoE is most visible in large language models, where it lets providers offer very large models at a workable running cost. Some well-known open models use eight experts per layer with two active at a time, which is where figures like “47 billion parameters, 13 billion active” come from: the total counts every expert, while the active count counts only what runs for one token, plus shared parts.
The idea is older than large language models, though. It grew out of work in the early 1990s on dividing a problem among local experts, and later work scaled it up for translation and other tasks. Today you may also meet it in research on vision and multimodal models, where different experts handle different kinds of input. In everyday use, you are unlikely to choose a MoE model directly. You meet it indirectly, as the design inside a chatbot, translation tool, or coding assistant that needs to stay fast while staying large.
What are the downsides?
MoE is not a free lunch, and its downsides matter when you judge claims about a model. First, memory use is high. Loading every expert takes space, which can make a MoE model awkward to run on a single modest device, even though its per-question computing cost looks low. Second, training and serving are more complex. Work has to be split across devices, experts have to stay balanced, and communication between devices adds overhead.
Third, quality is not guaranteed just because a model is sparse. A badly routed model can waste experts, drop tokens at capacity limits, or lean too heavily on a few favourites. A well-tuned dense model can still beat a poorly tuned MoE model on a given task. So when you compare models, look at what they actually do on tasks you care about, not only at their total parameter count.
Finally, MoE does not by itself make a model more truthful. An expert can still produce a confident-sounding wrong answer, just as any language model can. The routing trick changes how computation is organised, not whether the model knows something. Checking important answers against a reliable source remains good practice, whatever the architecture.
FAQs
Is mixture of experts the same as using several AI models at once?
No, not quite. Running several separate models and combining their answers is an ensemble. In MoE, the experts and router are parts of one trained model, and only the chosen experts run for each input.
Does a MoE model use all its parameters for every question?
No. That is the point of the design. Only the routed experts, plus shared layers, are active for a given token. All parameters still need to be stored, though, so memory use stays high.
Why do MoE models have two different parameter counts?
The larger number is the total across all experts. The smaller number is the active count: the parameters actually used for one token. Both numbers are honest, but they describe different things, so compare like with like.
Is MoE better than a dense model?
It depends on the task, the training, and the hardware. MoE can scale capacity at lower active cost, but it adds routing complexity and memory demands. A dense model may be simpler and better suited for smaller jobs.
Do I need to do anything different when using a MoE-based chatbot?
No. You use it in the same way, with the same care: ask clear questions, check important facts, and do not treat a fluent answer as proof. The architecture is mostly invisible to the end user.
Conclusion
Mixture of experts is a practical answer to a simple problem: how do you make a model bigger without making every question proportionally more expensive? By adding specialists and a router that picks among them, engineers can grow capacity while keeping the work per question in check. The trade-offs, high memory use, trickier training, and the need for balanced routing, are real, but so is the benefit.
If you want a practical next step, try this the next time you read a model announcement. Look past the headline parameter count and ask how many parameters are active per token, and what the model actually does well on tasks like yours. That habit will tell you far more than the total size alone, and it works whether the model under the hood is dense, sparse, or a mixture of experts. For background on the broader field, see our guide to artificial intelligence, our beginner guide to machine learning and deep learning, and our explainer on model pruning, another way of making models leaner. For deeper technical detail, see the Wikipedia overview of mixture of experts and Hugging Face’s mixture of experts explained.
