How Mixture of Experts (MoE) Scales Large Language Models
From dense transformer bottlenecks and sparse gating to top-k routing, expert capacity, and Mixtral parameter scaling
“How can an AI model possess 47 billion parameters but compute each token as fast and cheaply as a 13-billion parameter model?”
Dense language models activate every parameter for every single token, coupling compute costs to model capacity. Sparse Mixture of Experts (MoE) architectures decouple parameter count from inference compute. By replacing monolithic feed-forward blocks with parallel expert sub-networks and routing tokens to top-2 experts via a softmax gating network, MoE models achieve frontier performance while slashing per-token FLOPs by up to 75%.
- Why do dense transformers hit a computational scaling wall where training larger models becomes economically prohibitive?
- What is the structural difference between the self-attention layer and the feed-forward network (FFN) in an MoE transformer?
- How does a sparse gating router mathematically select the top-k experts for each individual token?
- What is 'routing collapse' and why must MoE models train with an auxiliary load-balancing loss?
- What is the Expert Capacity Factor and what happens when an expert's buffer overflows (token dropping)?
- Why does a 47B-parameter MoE model (like Mixtral 8x7B) require the GPU VRAM of a 47B model but run at the latency of a 13B model?
- How do router z-loss and noisy top-k gating stabilize FP16/BF16 training against exponential logit drift?
- Why are experts not purely thematic (e.g. 'the math expert' or 'the French expert'), and how do they actually specialize across syntactic and functional features?
To build a complete first-principles mental model of this mechanism, we recommend starting with these foundational explainers:
In a standard "dense" transformer—such as GPT-3, LLaMA-2, or Claude 2—every single parameter in the neural network is activated for every single token processed.
If a dense model has 70 billion parameters, generating the word "the" requires multiplying the token embedding through all 70 billion weights. Generating a 1,000-word essay forces the GPU cluster to churn through 70 × 109 × 2 × 1000 = 1.4 × 1014 floating-point operations (FLOPs).
This linear coupling between model capacity (how much knowledge the network can store) and inference compute (how many FLOPs it takes to evaluate one token) creates an economic and thermodynamic wall. Doubling a model's knowledge traditionally doubled the electricity, GPU hours, and server latency needed to serve every single user query.
In late 2023, Mistral AI released Mixtral 8x7B. On benchmarks, it matched or outperformed the dense 70-billion-parameter LLaMA-2 70B across coding, mathematics, and reasoning. Yet Mixtral generated text at six times the speed of LLaMA-2 70B and consumed only a fraction of the compute.
Mixtral achieved this by using a Sparse Mixture of Experts (MoE) architecture. While Mixtral holds 46.7 billion parameters in GPU memory, it activates only 12.9 billion parameters per token.
MoE decouples model capacity from compute cost. It breaks the monolithic neural network into a committee of parallel sub-networks, using a lightweight mathematical router to dispatch each token to only the two most qualified experts.
+----------------------------------------------------------------------------------------------------+
| THE DENSE vs. SPARSE MoE PARADIGM |
| |
| Dense Transformer Layer Sparse MoE Layer (e.g. Top-2 Routing) |
| ┌─────────────────────────────┐ ┌──────────────────────────────────────────────┐ |
| │ Multi-Head Self-Attention │ │ Multi-Head Self-Attention (Shared by all) │ |
| └──────────────┬──────────────┘ └──────────────────────┬───────────────────────┘ |
| ▼ ▼ |
| ┌─────────────────────────────┐ ┌─────────────────────────────┐ |
| │ Monolithic Feed-Forward │ │ Gating Router Network │ |
| │ (100% weights activated) │ │ Selects Top-2 Experts: [3, 7]│ |
| └─────────────────────────────┘ └──────────────┬──────────────┘ |
| ┌──────────┴──────────┐ |
| ▼ ▼ |
| ┌──────────────┐ ┌──────────────┐ |
| │ Expert 3 │ │ Expert 7 │ |
| │ (Active FFN) │ │ (Active FFN) │ |
| └──────────────┘ └──────────────┘ |
| (Experts 0, 1, 2, 4, 5, 6 remain dormant) |
+----------------------------------------------------------------------------------------------------+
1. Where Parameters Live in a Transformer
To understand where MoE introduces sparsity, we must examine the anatomical breakdown of weights inside a transformer layer.
As established in How Transformers Changed AI, every decoder transformer block consists of two primary sub-layers:
- Multi-Head Self-Attention (MHA): Responsible for routing information across different token positions in the context window.
- Feed-Forward Network (FFN): A position-wise multi-layer perceptron that processes each token independently.
The Weight Asymmetry
In modern LLMs using the SwiGLU activation function (such as LLaMA and Mistral), the FFN block consists of three weight matrices: a gate projection Wgate, an up-projection Wup, and a down-projection Wdown:
Where x ∈ Rdmodel, Wgate, Wup ∈ Rdmodel × dff, and Wdown ∈ Rdff × dmodel.
Typically, the intermediate dimension dff is set to roughly 83 dmodel.
When you sum the parameters across a layer:
- The attention layer (Wq, Wk, Wv, Wo) contains 4 × dmodel2 parameters.
- The SwiGLU FFN layer contains 3 × dmodel × dff ≈ 3 × dmodel × (83 dmodel) = 8 × dmodel2 parameters.
The Feed-Forward Network contains two-thirds (66.7%) of all parameters in the entire model.
Furthermore, researchers discovered that self-attention layers learn relational syntax (how words link to other words), while the FFN layers act as key-value associative factual memories—storing world knowledge, trivia, historical dates, and vocabulary definitions.
A dense model forces the token "the" to query the exact same massive factual memory matrices as the token "quantum" or "mitochondria".
MoE exploits this insight: keep the attention layers shared, but replace the monolithic FFN with a collection of smaller, parallel expert FFNs.
2. The Mathematical Anatomy of a Sparse MoE Layer
In an MoE layer, the single feed-forward block is replaced by N independent expert networks {E1, E2, …, EN}, governed by a Gating Network (Router) G(x).
Input Token Embedding x
│
┌─────────────────────┴─────────────────────┐
│ │
▼ ▼
┌────────────────────────┐ ┌────────────────────────┐
│ Gating Router │ │ N Parallel │
│ G(x) │ │ Expert FFNs │
└───────────┬────────────┘ │ [E_0, E_1, ..., E_N] │
│ └───────────┬────────────┘
▼ │
Softmax Logits │
[0.01, 0.65, ..., 0.28] │
│ │
▼ │
Top-k Gating Mask: │
Select Expert 1 (weight 0.70) │
Select Expert 7 (weight 0.30) │
│ │
└─────────────────────┬─────────────────────┘
▼
y = G_1(x) · E_1(x) + G_7(x) · E_7(x)
For an input token representation x ∈ Rdmodel, the output of the MoE layer is the weighted sum of the outputs of the selected experts:
Where:
- Ei(x) is the standard SwiGLU FFN output of the i-th expert.
- G(x)i is the routing probability weight assigned to expert i.
- Crucially, G(x) is sparse: for most experts, G(x)i = 0.
The Top-k Softmax Router
The simplest router is a parameterized linear projection matrix Wg ∈ Rdmodel × N. It projects the token vector into N logits (one per expert):
To enforce sparsity, the router selects only the k highest logits (typically k=1 in Switch Transformers or k=2 in Mixtral):
The gating weights are then normalized using a softmax function over the selected top-k experts:
For any expert not in the top k, the input to the softmax is −∞, yielding G(x)i = exp(−∞) = 0.
Because G(x)i = 0, the model never evaluates Ei(x). The sub-network is never loaded into GPU tensor cores, and zero multiply-accumulate FLOPs are spent on it.
Multi-head attention output vector x in R^d arrives at MoE block
Linear projection W_g computes affinity scores H(x) across all N experts
Router selects top-2 experts and normalizes their gating weights
Token is dispatched to Expert 3 and Expert 7 buffers (subject to capacity limits)
Expert 3 and Expert 7 compute independent SwiGLU forward passes
Outputs are multiplied by router weights G_3(x) and G_7(x) and summed
Combined expert output is added to residual stream x + MoE(x) and normalized
3. The Collapse Problem and Load-Balancing Losses
While the top-k gating formulation appears clean, naive training of an MoE network collapses into a catastrophic failure mode known as Router Collapse.
The Winner-Take-All Feedback Loop
In early training, random weight initialization causes one or two experts to perform slightly better on common tokens (like punctuation or stop words).
- The router sends more tokens to these favored experts.
- Because they receive more tokens, their weights receive more gradient updates, making them even more proficient.
- The router becomes even more confident, routing 99% of all tokens to the same two experts, while the remaining N−2 experts receive zero tokens and zero gradients.
The network devolves into an under-parameterized dense model, completely defeating the purpose of having multiple experts.
+──────────────────────────────────────────────────────────────────────────────────────────────────+
| ROUTER COLLAPSE DYNAMICS |
+──────────────────────────────────────────────────────────────────────────────────────────────────+
| Normal Balanced Routing: |
| Tokens ──► [Router] ──┬──► Expert 0 (12.5% of tokens) |
| ├──► Expert 1 (12.5% of tokens) |
| ├──► Expert 2 (12.5% of tokens) ... Perfect utilization across cluster |
| └──► Expert 7 (12.5% of tokens) |
| |
| Collapsed Routing (Without Auxiliary Loss): |
| Tokens ──► [Router] ──┬──► Expert 0 (95.0% of tokens) ◄─── OVERLOADED / OOM / BOTTLENECK |
| ├──► Expert 1 ( 5.0% of tokens) |
| ├──► Expert 2 ( 0.0% of tokens) ◄─── STARVED / DEAD WEIGHT |
| └──► Expert 7 ( 0.0% of tokens) ◄─── STARVED / DEAD WEIGHT |
+──────────────────────────────────────────────────────────────────────────────────────────────────+
The Auxiliary Load-Balancing Loss
To enforce an even distribution of tokens across all N experts, modern MoE models add a specialized auxiliary loss (Lbalance) to the main language modeling cross-entropy loss:
Where α is a weighting hyperparameter (typically α ∈ [0.01, 0.02]).
In the GShard (Lepikhin et al., 2020) and Switch Transformer (Fedus et al., 2022) formulation, the load-balancing loss is defined as the scaled dot product between the fraction of tokens dispatched to each expert and the average routing probability assigned to each expert across a batch:
Where:
- T is the total number of tokens in the batch.
- fi = 1T ∑x ∈ X I(Expert i ∈ TopK(x)) is the actual fraction of tokens dispatched to expert i.
- Pi = 1T ∑x ∈ X Softmax(H(x))i is the mean routing probability allocated to expert i.
Why Multiply fi by Pi?
The discrete selection function I(·) is non-differentiable (you cannot take a gradient through an argmax or top-k operation). However, Pi is a smooth, continuous softmax function of the router weights Wg.
Because Lbalance couples the actual token assignment fi with the differentiable probability Pi:
- If expert i is overloaded (fi is large), the loss heavily penalizes the router for assigning high probabilities Pi to that expert.
- Backpropagation pushes the router's logits downward for overloaded experts and upward for starving experts.
The loss reaches its theoretical global minimum when tokens are distributed with perfect uniformity: fi = 1N and Pi = 1N for all i.
Router z-Loss: Numerical Stability in Float16
In large-scale distributed training using 16-bit precision (FP16 or BF16), routers suffer from logit drift. Because the softmax function is translation-invariant (softmax(x) = softmax(x − c)), logits H(x)i can drift into extreme positive magnitudes (>104) without changing the softmax output probabilities. In FP16, this causes severe exponent overflow and NaN crashes.
To prevent logit explosion, Switch Transformers and PaLM-MoE introduce the Router z-loss:
This acts as an L2 regularizer on the log-partition function, penalizing large logits and ensuring numerical stability throughout months of continuous training.
4. Hardware Realities: Expert Capacity and Token Dropping
In theoretical machine learning, each expert can process an arbitrary number of tokens. In physical computer engineering, GPUs demand fixed, rectangular tensor shapes for maximum throughput.
If a GPU allocates a dynamic, unpredictable buffer size for every expert on every forward pass, GPU memory allocators fragment, CUDA kernels lose static compilation optimizations, and distributed clusters desynchronize.
To make MoE computationally efficient on NVIDIA H100 and Google TPU clusters, systems introduce the Expert Capacity Factor (C).
+──────────────────────────────────────────────────────────────────────────────────────────────────+
| EXPERT CAPACITY AND TOKEN DROPPING |
+──────────────────────────────────────────────────────────────────────────────────────────────────+
| Total Batch Tokens: 16 | Number of Experts: 4 | Capacity Factor C = 1.0 |
| Capacity per Expert = (16 / 4) * 1.0 = 4 tokens |
| |
| Expert 0 Buffer (Capacity = 4): [Token 1, Token 5, Token 9, Token 12] ◄── Full (100% capacity) |
| Expert 1 Buffer (Capacity = 4): [Token 2, Token 8] ◄── Underfull (Padded) |
| Expert 2 Buffer (Capacity = 4): [Token 3, Token 4, Token 7, Token 10] ◄── Full (100% capacity) |
| |
| Token 14 also routed to Expert 2! |
| Buffer is full! ──► [TOKEN DROPPING TRIGGERED] |
| Token 14 bypasses Expert 2 FFN; passes through residual connection completely untouched! |
+──────────────────────────────────────────────────────────────────────────────────────────────────+
The Capacity Formula
The Expert Capacity is the maximum number of tokens a single expert is permitted to process in a given batch:
Where C is the Capacity Factor:
- C = 1.0: Experts can process exactly their fair-share average number of tokens.
- C = 1.25: Experts have a 25% slack buffer to absorb statistical clustering.
- C = 2.0: Generous headroom, but requires allocating 2× larger memory buffers.
What Happens When Capacity Is Exceeded?
If an expert's capacity is 4 tokens, and the router attempts to send 6 tokens to that expert:
- The first 4 tokens enter the expert and are processed normally.
- The remaining 2 tokens are dropped.
When a token is dropped by an expert, it bypasses the FFN entirely. Its representation is simply passed through the residual stream unmodified (y = x).
While token dropping sounds destructive, language models are surprisingly resilient to occasional dropped tokens during pretraining. However, during inference, token dropping causes noticeable degradation. Modern models (like Mixtral and DeepSeek) set capacity high enough during inference—or use dynamically routed ragged tensors—to ensure zero token dropping in production.
5. Distributed Systems: Expert Parallelism and All-to-All
Training a dense model across multiple GPUs uses Tensor Parallelism (splitting weight matrices within a layer across GPUs) and Pipeline Parallelism (placing different layers on different GPUs).
MoE architectures introduce a third, unique axis of distributed scaling: Expert Parallelism (EP).
GPU 0 (Host) GPU 1 GPU 2 GPU 3
┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐ ┌─────────────────────┐
│ Shared Self-Attn │ │ Shared Self-Attn │ │ Shared Self-Attn │ │ Shared Self-Attn │
│ Router Network │ │ Router Network │ │ Router Network │ │ Router Network │
├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤ ├─────────────────────┤
│ Expert 0 | Expert 1 │ │ Expert 2 | Expert 3 │ │ Expert 4 | Expert 5 │ │ Expert 6 | Expert 7 │
└──────────┬──────────┘ └──────────┬──────────┘ └──────────┬──────────┘ └──────────┬──────────┘
│ │ │ │
└─────────────────────────┴────────────┬────────────┴─────────────────────────┘
▼
Distributed All-to-All Dispatch
(NVLink / InfiniBand Bus)
Token A (on GPU 0) routed to Expert 5 (on GPU 2)
Token B (on GPU 3) routed to Expert 1 (on GPU 0)
In Expert Parallelism:
- The attention layer and router are replicated across all GPUs.
- The N experts are sharded across the cluster. If you have 8 GPUs and 8 experts, each GPU hosts exactly one expert.
The All-to-All Communication Bottleneck
Consider GPU 0 processing a batch of tokens:
- GPU 0 computes self-attention and router logits for its local tokens.
- The router determines that Token A needs Expert 5. But Expert 5 lives on GPU 2!
- All GPUs in the cluster must execute an All-to-All Collective Communication (via MPI or NCCL):
- Every GPU transmits tokens destined for remote experts across high-speed InfiniBand or NVLink interconnects.
- Every GPU receives tokens from remote GPUs destined for its local expert.
- Each GPU runs the forward pass of its local expert on the gathered tokens.
- A second All-to-All communication sends the expert outputs back to the originating GPUs to be summed and added to the residual stream.
This reveals the fundamental engineering trade-off of MoE:
If GPUs are connected via ultra-fast NVLink (900 GB/s on H100), the communication overhead is negligible compared to the massive compute savings. But if run over slow consumer PCIe or Ethernet, the all-to-all communication latency obliterates any speed advantage from sparsity.
6. The Mixtral 8x7B Case Study
To see how these principles unite in a production frontier model, let us examine the architecture of Mixtral 8x7B (Mistral AI, December 2023).
Parameter Activation per Token
Dense model activates 100% of parameters on every token; MoE activates only 15-25% of parameters per token.
Training Compute Efficiency
Dense models require full FLOP scaling for larger capacity; MoE achieves lower pretraining loss with 2-4x fewer training FLOPs.
Inference FLOPs vs Latency
Dense 70B model requires 70B FLOPs per token; Mixtral 8x7B (47B total) executes only 13B FLOPs per token for 3x faster generation.
GPU VRAM Memory Footprint
Dense 13B model fits in 24 GB VRAM; MoE 8x7B still holds all 47B weights in memory, requiring ~90 GB VRAM.
Hardware Communication Overhead
Dense model has minimal cross-GPU sharding overhead; MoE requires fast inter-GPU all-to-all communication across expert ranks.
+───────────────────────────┬───────────────────────────────────┬──────────────────────────────────+
| Architectural Metric | LLaMA-2 70B (Dense Model) | Mixtral 8x7B (Sparse MoE Model) |
+───────────────────────────┼───────────────────────────────────┼──────────────────────────────────+
| Total Parameters | 68.9 Billion | 46.7 Billion |
| Active Parameters / Token | 68.9 Billion (100%) | 12.9 Billion (27.6%) |
| Number of Experts | 1 (Monolithic FFN) | 8 Experts per Layer |
| Experts Selected (Top-k) | N/A | k = 2 Experts per Token |
| Hidden Dimension (d_model)| 8,192 | 4,096 |
| Number of Layers | 80 Layers | 32 Layers |
| Context Window | 4,096 tokens | 32,768 tokens |
| Inference Speed (Tokens/s)| ~15 tokens/sec (Single node) | ~65 tokens/sec (Single node) |
| Performance Benchmarks | Baseline | Matches or exceeds LLaMA-2 70B |
+───────────────────────────┴───────────────────────────────────┴──────────────────────────────────+
Why is Mixtral Called "8x7B" if it Only Has 47B Parameters?
A common misconception is that Mixtral simply took eight complete 7B models and glued them together (8 × 7 = 56B).
In reality:
- The self-attention layers are completely shared across all experts.
- The embedding matrices and RMSNorm layers are completely shared.
- Only the Feed-Forward Networks (FFNs) are duplicated eight times.
Because the FFN represents roughly two-thirds of a transformer's weights, duplicating the FFN eight times across 32 layers results in 46.7 billion total parameters, not 56B.
When processing any token:
- It passes through the shared attention layer (accounting for ≈ 3.3B active parameters).
- It is routed to exactly two of the eight FFN experts (2 × ≈ 4.8B = 9.6B active parameters).
- Total active parameters per token = 3.3B + 9.6B = 12.9 Billion.
Mixtral achieves the benchmark reasoning power of a 70B dense model while paying the compute cost of a 13B model on every token!
7. What Do Experts Actually Learn?
When the MoE architecture was first conceived, researchers hypothesized that experts would naturally partition by human domains: one expert would become the "coding expert", another the "history expert", and another the "French language expert".
Empirical analysis of trained routers (including Mixtral and GShard) shattered this romantic notion.
Expert Assignment Distribution by Linguistic Feature
─────────────────────────────────────────────────────
Token: " def" ──► Expert 2 (Weight 0.65), Expert 6 (Weight 0.35) [Syntax / Code indentation]
Token: " calculate" ──► Expert 1 (Weight 0.80), Expert 4 (Weight 0.20) [Imperative Verbs]
Token: " 3.14159" ──► Expert 2 (Weight 0.55), Expert 7 (Weight 0.45) [Numerical Constants]
Token: " however" ──► Expert 0 (Weight 0.75), Expert 3 (Weight 0.25) [Conjunctive Adverbs]
Rather than high-level semantic domains, experts specialize in low-level grammatical, syntactic, and structural primitives:
- Punctuation and Whitespace Experts: Certain experts fire almost exclusively on Python indentation, newlines, quotation marks, and brackets.
- Syntactic Category Experts: Other experts specialize in coordinating conjunctions ("and", "but", "or"), relative pronouns, or auxiliary verbs.
- Numerical and Symbolic Experts: Specific experts activate for arithmetic digits, currency symbols, and mathematical operators.
- Contextual Polysemy: The word "Apple" will route to one expert when followed by "iPhone" or "NASDAQ", and to a completely different expert when followed by "cider" or "orchard".
The router performs high-speed, continuous feature decomposition. By allowing two experts to cooperate on each token, the model can blend syntactic handling (e.g. Expert 2 handling Python syntax) with domain knowledge (Expert 7 handling mathematical logic).
8. Summary: The First-Principles Architecture of Sparse MoE
Sparse Mixture of Experts represents a pivotal paradigm shift in scaling artificial intelligence:
- Decoupling Compute from Capacity: Dense models scale FLOPs linearly with parameter count. MoE models scale memory capacity while keeping per-token FLOPs constant.
- FFN Replacement: MoE leaves attention layers shared across the entire model, replacing only the parameter-heavy Feed-Forward Networks with parallel expert sub-networks.
- Top-k Gating: A linear router projects token representations to expert logits, selecting the top-k experts via a masked softmax and ignoring dormant experts entirely.
- Auxiliary Balancing Loss: Without load balancing, routers collapse into a winner-take-all feedback loop. Auxiliary losses penalize non-uniform token-to-expert distributions to ensure balanced GPU utilization.
- Hardware Capacity Limits: GPUs require static tensor shapes. Expert capacity factors define fixed memory buffers, dropping excess tokens through residual connections if buffers overflow.
- All-to-All Trade-Off: Expert Parallelism shards experts across GPUs, trading saved arithmetic FLOPs for inter-GPU communication bandwidth.
- Empirical Specialization: Experts do not divide neatly into academic subjects; they specialize across grammatical roles, syntactic structures, and functional sub-tasks.
Where to Go From Here
Follow downstream mechanisms, contrasting architectures, or applications of these principles.
Why AI Chatbots Sometimes Make Things Up
Deep-dive following foundational explainer Why AI Chatbots Sometimes Make Things Up
How AI Agents Work
More in Computing & AI
How an AI Agent Decides Which Tool to Use
More in Computing & AI
Why Neural Networks Were Invented
More in Computing & Artificial Intelligence
Verified Specifications & Architectural References
This explainer is grounded in primary-source engineering specifications, regulatory circulars, and standard documentation.