MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelism

Token dispatch communication (why layout matters)

Per MoE layer, tokens are sent to their experts and the results are sent back. A rough count of the bytes moved is:

where:

• $T$ is tokens in the microbatch,
• $k$ is top-$k$ experts per token,
• $d_{\mathrm{model}}$ is hidden width,
• $b$ is bytes per element (e.g., 2 for fp16),
• the factor of 2 is dispatch + combine.

The traffic grows with $k$, not with the number of experts: going from top-1 to top-2 routing doubles it. This rough count also assumes every token leaves its GPU; tokens whose expert sits on the same GPU move nothing. If communication is the bottleneck, spreading experts over more GPUs or slower links can make the layer slower even though each token's expert computation is small.

Stragglers from routing skew

Let $n_e$ be the number of tokens routed to expert $e$ in a microbatch. If expert compute time is proportional to token count, then layer time is dominated by:

Even if the average load is small, the maximum load can be much larger when routing is skewed. Every token in the layer waits for the busiest expert, so one overloaded expert slows the whole batch.

Disaggregation (separate pools)

One recent response is disaggregation: run attention on one pool of GPUs and the experts on another, and pipeline microbatches between them so that both pools stay busy (Zhu et al., 2025). Each pool can then be sized and parallelized for its own work, at the cost of extra communication between the pools.

Worked example: only served, remote assignments move

The count $2\cdot T\cdot k\cdot d_{\mathrm{model}}\cdot b$ assumes that every token's $k$ choices are served and that every one leaves its GPU. Neither need hold. Take $T=4$ tokens on GPU D0, top-2 routing over three experts, and a capacity of two tokens per expert, filled in token order:

• T0 and T1 choose E0 and E1, and both are served;
• T2 chooses E0 and E2, but E0 is full, so only E2 serves it;
• T3 chooses E1 and E2, but E1 is full, so only E2 serves it.

Six of the eight choices are served, two per expert; the two that overflow move nothing (the mixture-of-experts page works through capacity). Now place E1 on D0 and E0 and E2 on D1. A served assignment is remote when its expert's GPU differs from its token's GPU, and each remote assignment sends one activation out and receives one result back. Four are remote (T0→E0, T1→E0, T2→E2, T3→E2), so with $d_{\mathrm{model}}=4096$ and $b=2$,

half of the rough count, $2\cdot4\cdot2\cdot4096\cdot2=131{,}072$ bytes. Move only E2 onto D0: the same six assignments are served and every expert still holds two tokens, but only T0→E0 and T1→E0 are remote, and the payload halves to $32$ KiB.

These are the bytes the dispatch and return must carry, not measured traffic. Real systems pad and pack buffers, add metadata, choose different collective operations, overlap communication with computation and pay a fixed cost per message, so neither the bytes nor their halving translate directly into time.