Mixture of Experts DeepSeek: How MoE Models Work

I checked the public DeepSeek-V3 technical report because it gives an inspectable Mixture of Experts (MoE) example: the model has 671 billion main-model parameters, but its active path uses 37 billion parameters for each token. DeepSeek can keep a large parameter pool without using the whole pool for every token.

A router chooses experts for each token

A Transformer block contains an attention stage and a feed-forward network (FFN). A dense model sends every token through the same FFN parameters, while a sparse MoE layer replaces that FFN with several smaller networks called experts.

Selected expert outputs are multiplied by routing weights and combined before the token continues through the model. The router first scores the available experts separately for each token and selects a small subset.

Routing happens at token level, so two tokens in one prompt can take different expert paths and the same expert can process tokens from many subjects.

DeepSeek-V3 activates eight routed experts per token

The DeepSeek-V3 technical report documents a concrete DeepSeekMoE configuration whose first three feed-forward layers stay dense and whose remaining feed-forward layers use MoE.

DeepSeek-V3 detail	Published configuration	What it means
Main-model parameters	671 billion	Total capacity stored in the model
Activated parameters	37 billion per token	The active computation path is much smaller than the total model
Shared experts	1 per MoE layer	Every token passes through this expert
Routed experts	256 per MoE layer	The router has a large expert pool to choose from
Selected routed experts	8 per token	Only a small subset runs for that token

The 37-billion figure covers the active model path, so it should not be read as the sum of eight routed experts alone. Attention layers, dense layers, embeddings, and the always-active shared expert also contribute to the computation.

Shared and routed experts have different jobs

The shared expert receives every token and can learn knowledge that many token types need. Routed experts divide the remaining feed-forward work, which gives the model more parameter capacity without applying every expert to every token.

Names such as “science expert” or “finance expert” are teaching shortcuts, not labels assigned by DeepSeek. Expert roles emerge during training, and their learned behavior can reflect syntax, concepts, or token combinations that do not fit a human subject category.

The routing path step by step

One token passes through a DeepSeekMoE layer using the following sequence:

1. The layer receives the token’s hidden-state vector.
2. The router calculates an affinity score for each routed expert.
3. Routing selects eight experts using the top scores after the training-time balancing adjustment.
4. The selected experts process the token in parallel, and the shared expert processes it too.
5. The layer combines expert outputs using gating weights derived from the original affinity scores.

Load balancing keeps experts available

A router can collapse toward a few preferred experts during training. Those experts become overloaded, other experts learn too little, and distributed hardware waits for the busiest route.

DeepSeek-V3 adjusts a bias attached to each expert’s routing score. Training lowers the bias for an overloaded expert and raises it for an underused one, which changes expert selection without changing the gating weight applied to the selected expert’s output.

The report calls this an auxiliary-loss-free load-balancing strategy because the main balancing signal does not require a conventional global auxiliary loss. DeepSeek-V3 still uses a very small sequence-level balance loss to prevent extreme imbalance inside one sequence.

Sparse activation changes compute, not storage

MoE reduces the parameters used for one token relative to activating the entire model, but the full expert pool still has to be stored, loaded, and distributed across accelerators, and token routing creates communication between devices.

Memory bandwidth, batch size, expert placement, and network traffic affect serving speed, so sparse activation does not guarantee lower latency on every deployment. A larger total parameter count also does not translate directly into the same inference cost as a dense model of equal size.

How MoE relates to DeepSeek-R1

DeepSeek-R1 and DeepSeek-R1-Zero were trained from DeepSeek-V3-Base, placing the MoE architecture underneath their reasoning training and connecting it to the reinforcement-learning stage covered in our DeepSeek-R1 explanation.

If you want to call a hosted model rather than inspect its expert routing, follow the current setup in our DeepSeek API guide and verify the model name against DeepSeek’s API documentation before sending requests.

A useful way to read MoE claims

Check four numbers whenever a model is described as MoE: total parameters, activated parameters per token, the number of routed experts, and how many experts each token selects. Those values tell you more than the word “sparse” on its own.

Then check whether the source discusses load balancing and communication cost. Sparse activation explains the computation path, but deployment efficiency depends on how well the router and hardware execute that path.

Questions about DeepSeek MoE