Mixture of experts

A dense language model uses every one of its parametersParameterA number inside a model that training adjusts, such as the slope of a line or a weight in a neural network. Large language models have billions of them.Taught in The learning loopOpen in glossary for every token it reads or writes. That gets expensive quickly. A mixture-of-expertsMixture of expertsAn architecture with many alternative sub-networks (experts) and a router that runs only a few of them for each token, so a model can store many parameters while using few per token.Open in glossary (MoE) model takes a different approach: it holds many alternative sub-networks, called experts, and for each token it runs only a few of them. The model can store far more knowledge than it spends computation on.

The idea is old (Jacobs and colleagues described mixtures of local experts in 1991) and was scaled up for deep networks by Shazeer et al. (2017) with a sparsely gated layer. Many recent large models are built this way, including Mixtral 8x7B, DeepSeek-V3, and, by Google’s account, its Gemini models; the 2026 section below shows what the published ones look like.

Where the experts live

In a transformer, each layer has an attention block followed by a feed-forward network. In an MoE transformer the attention block stays as it is, and the single feed-forward network is replaced by several experts, each a feed-forward network of its own, plus a small routerRouterThe small learned layer in a mixture-of-experts model that scores every expert for each token and picks which few experts will process it.Open in glossary.

For a token with hidden vector $x$, the router computes one score per expert, $s = W_r\, x$. The top $k$ scores pick the experts that will run. Their outputs are mixed with gate weights $g_i$ that are a softmax over just the chosen scores:

This is how Mixtral combines its experts. Experts outside the top $k$ contribute nothing and cost nothing for that token.

import numpy as np

def softmax(z, axis=-1):
    e = np.exp(z - z.max(axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

def show(values):
    return " ".join(f"{v:.3f}" for v in values)

# Router scores s = W_r x for one token, over 8 experts.
scores = np.array([0.4, 2.1, -0.3, 1.2, 1.7, -1.0, 0.0, 0.6])
probs = softmax(scores)          # the router's probability for each expert
top = np.argsort(-scores)[:2]    # top 2: only these experts run
gates = softmax(scores[top])     # softmax over the chosen scores only
print("probabilities:", show(probs))
print("experts that run:", ", ".join(f"E{e + 1}" for e in top))
print("gate weights:", show(gates), "| renormalized:", show(probs[top] / probs[top].sum()))

# Load over a batch: four tokens, four experts, top 1.
scores = np.array([[2.0, 0.1, -0.5, 0.3],
                   [1.5, 0.2, 0.0, -1.0],
                   [1.8, -0.2, 0.4, 0.1],
                   [0.1, 1.2, 0.3, -0.4]])
chosen = scores.argmax(axis=1)
f = np.bincount(chosen, minlength=4) / len(chosen)  # share of tokens per expert
P = softmax(scores, axis=1).mean(axis=0)            # mean router probability
print("tokens per expert:", np.bincount(chosen, minlength=4).tolist())
print(f"balance term N * sum(f * P) = {4 * np.sum(f * P):.3f} (1.0 is perfectly even)")

probabilities: 0.067 0.365 0.033 0.148 0.245 0.016 0.045 0.081
experts that run: E2, E5
gate weights: 0.599 0.401 | renormalized: 0.599 0.401
tokens per expert: [3, 1, 0, 0]
balance term N * sum(f * P) = 1.834 (1.0 is perfectly even)

Keeping the load balanced

Left alone, a router tends to favor a few experts. Those experts improve faster because they see more tokens, so the router favors them even more, and the rest wither. MoE training adds an auxiliary loss to push against this. A common form, from the Switch Transformer (Fedus et al., 2022), is

where $N$ is the number of experts, $f_i$ is the fraction of token assignments that went to expert $i$, $P_i$ is the router’s average probability for expert $i$, and $\alpha$ is a small coefficient. Without $\alpha$, the sum is 1 when routing is perfectly even and grows as tokens crowd onto a few experts. The demo shows it as the balance term. Some systems also cap how many tokens one expert can take per batch.

The auxiliary loss has a cost: its gradients can work against the main training objective, and DeepSeek’s V3 report notes that “too large an auxiliary loss will impair the model performance”. Two later fixes are now in use:

• A bias that steers routing. DeepSeek-V3 (DeepSeek-AI, 2024), following Wang et al. (2024), adds a bias $b_i$ to each expert’s score only when choosing the top $k$; the gate weights still come from the original scores. After each training step, every overloaded expert’s bias goes down by a small amount $\gamma$ (0.001 for most of its training) and every underloaded expert’s goes up, so traffic evens out without adding any gradient. A very small balance loss within each sequence stays as a backstop, and no tokens are dropped. Kimi K2, GLM-5, and DeepSeek-V4 list the same method in their configuration files.
• Balance over a bigger batch. Computed over each small slice of a batch, the balance term pushes every single text to spread its tokens over all experts, even a text on a single subject. Qiu et al. (2025) compute it over the whole global batch instead, so each text can concentrate on a few experts while the total stays even, and report better models with more specialized experts. Qwen3 adopted it (Qwen team, 2025).

A real router

The router above is random. The one below is real. OLMoE-1B-7B is an open model from the Allen Institute for AI, trained from scratch on about 5 trillion tokens. Each of its 16 layers has 64 small experts, and 8 of them run for each token, so it stores 6.9B parameters but uses about 1.3B for each token. We ran it over six short texts and recorded, for every token at every layer, the router’s probability for each expert and the 8 experts that ran.

Total parameters versus active parameters

Because only $k$ of $N$ experts run, an MoE model has two sizes. The total count is everything stored. The active count is what one token actually passes through: all the shared parts plus $k$ experts per layer.

# Mixtral 8x7B's published shape.
d, d_ff, layers, heads, kv_heads, vocab = 4096, 14336, 32, 32, 8, 32000

def count(experts, k):
    expert = 3 * d * d_ff  # one SwiGLU expert: three weight matrices
    attention = 2 * d * d + 2 * d * kv_heads * (d // heads)  # Q and O, then K and V
    router, norms = d * experts, 2 * d
    shared = layers * (attention + router + norms) + 2 * vocab * d + d
    return shared + layers * experts * expert, shared + layers * k * expert

for experts, k in ((1, 1), (8, 2), (64, 2)):
    total, active = count(experts, k)
    name = f"{experts} expert" + ("s" if experts > 1 else "")
    print(f"{name}, top {k}: {total / 1e9:.1f}B stored, "
          f"{active / 1e9:.1f}B per token, {2 * total / 1e9:.0f} GB at 16 bits")

1 expert, top 1: 7.2B stored, 7.2B per token, 14 GB at 16 bits
8 experts, top 2: 46.7B stored, 12.9B per token, 93 GB at 16 bits
64 experts, top 2: 362.4B stored, 12.9B per token, 725 GB at 16 bits

The name “8x7B” suggests 56B parameters, but only the feed-forward blocks are copied eight times. Attention, embeddings, and normalization are shared, so the total is about 47B. For each token, the model does roughly the arithmetic of a 13B dense model.

What mixture of experts buys, and what it costs

Compute. Training and inference arithmetic per token scale with active parameters, so an MoE model can be trained on more data for the same budget, or be larger for the same speed.

Memory. Every expert must be stored, and typically held in accelerator memory, because the next token might need any of them. Mixtral’s 46.7B parameters take about 93 GB at 16 bits per weight, even though each token uses only 12.9B of them.

Engineering. Tokens for different experts must be sent to wherever those experts live, often across many chips. Load imbalance turns directly into idle hardware.

Mixture of experts in 2026

Many of the large open-weight language models released in 2025 and 2026 are mixtures of experts. Their published shapes, as of October 2026, from each model’s card and configuration file:

“Active” counts the parameters each token uses, “Experts per layer” counts the routed experts, “Chosen” is how many of them run for each token, and “Shared” counts experts that run for every token. Kimi K2 is from Moonshot AI, gpt-oss from OpenAI, and Qwen3.5 from Alibaba. Sources: the Mixtral paper, the DeepSeek-V3 report, and the model cards for Kimi K2, gpt-oss, Qwen3.5, and DeepSeek-V4.

Three changes since Mixtral stand out.

Many small experts. DeepSeekMoE (Dai et al., 2024) split each expert into several smaller ones and turned on proportionally more of them, for “a more flexible combination of activated experts”. Choosing 2 of 8 experts allows 28 combinations in each layer; choosing 8 of 256 allows about $4.1 \times 10^{14}$.

An expert every token uses. Most of these models also have a shared expertShared expertIn a mixture-of-experts layer, an expert that runs for every token alongside the few the router picks, meant to hold what all tokens need so the routed experts can specialize.Open in glossary that runs for every token beside the routed ones, “aiming at capturing common knowledge and mitigating redundancy in routed experts” (Dai et al.). It is not settled practice: Qwen3 left shared experts out, and Qwen3.5 has one.

A smaller active share. Mixtral uses 28% of its parameters for each token; the later models in the table use about 3 to 6%. Each token costs about as much arithmetic as a dense model of 5 to 50 billion parameters, while memory still follows the total. DeepSeek-V4-Pro’s 1.6 trillion parameters would take 3.2 TB at 16 bits per weight, so DeepSeek released it with the expert weights stored at 4 bits and most other weights at 8. OpenAI quantized gpt-oss-120b’s expert weights to a 4-bit format, MXFP4, so that the model fits on a single 80 GB GPU.

The routers differ in detail too. Mixtral and gpt-oss take a softmax over the chosen experts’ scores, OLMoE over all of them; DeepSeek-V3, Kimi K2, and GLM-5 score experts with a sigmoid and rescale the chosen ones; DeepSeek-V4 changes the scoring function again (technical report) and, in its first three layers, picks experts with a fixed hash function instead of learned scores.

Closed models publish less. Google’s Gemini 2.5 report and its Gemini 3 Pro model card describe them as sparse mixtures of experts, which “decouple total model capacity from computation and serving cost per token”, but give no expert counts or sizes.