Mixture of Experts, Playable

EP 01

The router — each token picks its own specialists

In a dense model, every token flows through the same giant network. In a Mixture-of-Experts layer, a tiny router looks at each token and sends it to just 2 of 8 experts. Different kinds of tokens learn to prefer different experts — code goes one way, French goes another. Click token types and watch the router light up the path.

Pick a token type above.

Conceptual router: real MoE routers are learned linear layers producing a score per expert, then a top-2 softmax. The specialization (code→one expert, French→another) genuinely emerges during training.

Why it matters: the router is the whole trick. It lets the model own a huge library of specialists while only ever paying to run two of them per token — this is how Mixtral, DeepSeek-V3 and GPT-scale MoE models work.

EP 02

Huge on paper, cheap in practice — the params calculator

This is the number that makes headlines. Set the expert count, the params per expert, and how many the router activates per token. The total is what you must store; the active is what you pay to compute for each token. Drag the sliders and watch the two numbers diverge.

Experts (N) 8 Params / expert (B) 7 Active experts (k) 2

Shared attention/embedding params are ignored here to keep the mental model clean — the expert FFN blocks are where the total-vs-active gap lives, and that's the point.

The one-liner: capacity scales with total params, compute cost scales with active params. Crank N to 64 and you're storing a giant brain but still only computing a tiny slice per token. That decoupling is why MoE exists.

EP 03

Capacity almost for free — MoE vs a dense twin

Here's the good case, head to head. A dense model that wanted the same knowledge capacity would have to run all of it on every token. An MoE gets roughly the same capacity while computing only its active slice. Feed both the same token stream and compare quality-per-token against compute-per-token.

Both models start idle.

Capacity ≈ how much specialized knowledge the model can hold. The MoE here has 8× the experts of the dense twin's single block, at ~1.1× the per-token FLOPs (router overhead). Numbers are computed live from the run below.

Why it matters: this is the free-lunch that isn't quite free — you trade cheap memory (store all experts) for expensive compute savings. It only pays off when your traffic actually uses the whole committee, which EP 04 shows can fail.

EP 04

The hot expert — break it with load imbalance

The router is free to send tokens wherever it likes — and left alone, it plays favorites. One "hot" expert gets swamped while others sit idle. Overloaded experts hit a capacity limit and start dropping tokens (those tokens skip the layer entirely). Skew the traffic and watch the drops climb, then turn on load balancing to fix it.

Traffic skew 70%

Conceptual: real systems add an auxiliary load-balancing loss during training plus a capacity factor at inference. Dropped-token counts and the imbalance ratio below are computed live from the routing you set.

The failure: without balancing pressure, MoE collapses toward using a few experts — wasting the capacity you paid to store, and silently dropping tokens. Half the engineering of a real MoE is fighting this. Flip balancing on and watch the load flatten.

EP 05

Routing flips — why near-identical inputs can diverge

Top-2 routing is a hard, discrete choice: an expert is either in or out. When two experts score almost the same, a microscopic change to the input — one extra word, a rounding difference — can flip which expert wins, sending the token down a different path and changing the output. Nudge the input near a decision boundary and catch the flip.

Input nudge -0.40

Conceptual: two experts' router scores are drawn as lines that cross. Away from the crossing, tiny nudges do nothing (good case). Right at it, a 0.01 nudge flips the chosen expert (bad case) — a real, documented source of MoE non-determinism, made worse by batch effects.

Why it matters: the same prompt can route differently depending on what else is in the batch, so MoE models are famously harder to make bit-for-bit reproducible. Discrete routing buys efficiency but sells away smoothness — that's the trade at the heart of the architecture.