Continuous Function

Foundation reference

MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelism

In MoE inference the busiest expert sets each layer's latency, so uneven routing slows the whole batch.

A newer notebook covers this concept. This older page keeps its summary, math and sources.

Concept 23 of 100EfficiencyStudy phase 6
Open the demoOpen the newer notebookAll concepts
Key equation
tlayer≈max⁡ec⋅ne⋅dmodel⋅dffnFLOPSet_{\text{layer}} \approx \max_{e} \frac{c \cdot n_e \cdot d_{\text{model}} \cdot d_{\text{ffn}}}{\text{FLOPS}_e}
Reading map and next steps

Study guide

Where to start

Concept 23 of 100MoE ServingEfficiency · Study phase 6: Modern efficiency & inference
Question

Why does sparsity trade arithmetic for a scheduling problem, in which load is bursty and uneven and the busiest expert sets the latency?

Open the demoGo to the mathOpen the newer notebookNext concept
How to study this pagePredict · Check · Generalize · Continue
PredictSay what you expect.

Before you move a control in the demo, say what you expect to change.

CheckCheck against the math and the papers.

Compare what the demo shows with the key equation and with the papers cited below.

GeneralizeName what stays the same.

Identify the part of the idea that survives a change of notation or model, so you can recognize it in another paper or system.

ContinueContinue in the newer notebook.

The newer notebook covers the same concept and has an interactive demo.

Why It Matters for Modern Models

  • In MoE inference the busiest expert sets each layer's latency, so uneven routing slows the whole batch.
  • With experts spread across devices, every MoE layer needs two all-to-all exchanges (dispatch and combine), whose cost is set by network bandwidth rather than arithmetic.
  • MegaScale-Infer reports up to 1.90× higher throughput per GPU than earlier systems by running attention and expert FFNs on separate GPUs (Zhu et al., 2025).
  • These constraints apply to any served MoE model, such as Mixtral or DeepSeek-V2, and explain many of their deployment choices.
  • This page combines LLM Serving and MoE: it covers what changes when a routed model is served to many users at once.

Questions to Think About

  • Why does each MoE layer need two collective communications (dispatch and combine) in addition to its matrix multiplications?
  • How many tokens must each expert receive for its matrix multiplications to run efficiently during decoding, and what limits the batch size (KV-cache memory, latency targets)?
  • What does expert parallelism change about scaling: expert placement, routing-induced traffic and micro-batching, rather than only tensor parallelism?
  • When is it worth placing attention and expert FFNs on separate GPUs, and how does pipelining micro-batches keep both pools busy?

Interactive Demos

Core Math

If you prefer intuition first, try the demos above, then come back for the math.

Key Equation
tlayer≈max⁡ec⋅ne⋅dmodel⋅dffnFLOPSet_{\text{layer}} \approx \max_{e} \frac{c \cdot n_e \cdot d_{\text{model}} \cdot d_{\text{ffn}}}{\text{FLOPS}_e}

Serving an MoE model turns sparse computation into a scheduling problem: routing decides how much work and communication each device gets.

Token dispatch (all-to-all exchange):

In each MoE layer, every token is sent to its kk chosen experts and the results are sent back. For a batch of TT tokens:

Bytescomm≈2⋅T⋅k⋅dmodel⋅b\text{Bytes}_{\text{comm}} \approx 2 \cdot T \cdot k \cdot d_{\text{model}} \cdot b

where dmodeld_{\text{model}} is the hidden dimension and bb the bytes per element; the factor 2 counts dispatch and combine. This is an upper bound reached when every routed token leaves its device.

Stragglers (routing skew):

When routing is uneven, the layer waits for the busiest expert:

tlayer≈max⁡ete=max⁡ec⋅ne⋅dmodel⋅dffnFLOPSet_{\text{layer}} \approx \max_{e} t_e = \max_e \frac{c \cdot n_e \cdot d_{\text{model}} \cdot d_{\text{ffn}}}{\text{FLOPS}_e}

where nen_e is the number of tokens routed to expert ee, FLOPSe\text{FLOPS}_e is the achieved floating-point throughput of the device holding it, and cc is 4 for an expert with two weight matrices (6 for a gated expert with three), at 2 operations per multiply-add. Even when the average load is low, the busiest expert sets the layer time and the other experts sit partly idle.

Disaggregation:

MegaScale-Infer places attention and expert FFNs on separate GPU pools, sized independently, and splits each batch into micro-batches that alternate between the two pools so that neither waits for the other (Zhu et al., 2025). The cost is extra communication between the pools.

Key Papers

MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism

Zhu et al.2025arXiv
Read paper →

Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling

Li et al.2024arXiv
Read paper →

Mixtral of Experts

Jiang et al.2024arXiv
Read paper →

Connections

Prerequisites

AttentionEfficient AttentionLLM ServingMoE

Where to go next

A newer notebook covers MoE Serving, with an interactive demo.

Continue in the newer notebook
Other connections (5)

Which limitations does it address, or leave to a later idea?

  • MoE scheduling: a limitation of MoE that MoE Serving addresses. Open MoE

    MoE makes compute sparse but introduces dispatch/combine all-to-all traffic and severe load skew. MoE serving systems were invented to schedule routing, reduce stragglers, and manage expert pools efficiently.

Where does it break down?

  • Routing stragglers: where MoE breaks down, taken up on this page. Open MoE

    Mixture-of-experts routing gives experts uneven loads, and the busiest expert sets the latency of the whole layer. Without careful scheduling and load balancing, the saving from sparse computation is lost to waiting on the slowest expert.

Where else does the same idea appear?

  • Hide latency with parallelism: the same technique appears in Speculative Decoding. Open Speculative Decoding

    MoE serving overlaps the all-to-all communication of routed tokens with computation, for example by pipelining micro-batches (Zhu et al., 2025); speculative decoding checks several drafted tokens in one parallel pass to shorten a sequential chain of steps. Both use parallel work to hide a sequential cost.

  • Memory paging strategies: the same technique appears in Long Context. Open Long Context

    Both MoE expert caching and long-context KV caching use similar memory management: page in/out based on access patterns, predict future needs, trade latency for capacity. PagedAttention borrows from OS memory management.

  • Scheduling under skew: an analogy between MoE Serving and LLM Serving. Open LLM Serving

    Both are scheduling problems under memory and latency constraints; MoE serving is like LLM serving with an extra routing-induced permutation and heavier tail-latency from skew.