Continuous Function

Muon Optimizer

Muon (Jordan et al., 2024) orthogonalizes the momentum update for a hidden layer’s two-dimensional weight matrix, such as a transformer’s linear or attention projection. We use a 2 × 2 update to compare the ideal orthogonalized matrix with five steps of a Newton–Schulz iteration; the Python code checks their singular values. Only the update is orthogonalized; the weights themselves are not constrained to be orthogonal. Other parameters, including embeddings, the output layer, biases and normalization gains, are usually optimized with AdamW. You will need matrix multiplication and singular value decomposition.

High-Level Idea

For a weight matrix W∈Rm×nW \in \mathbb{R}^{m \times n}, Muon makes each update in two steps:

  1. Momentum. Accumulate the gradient into a momentum matrix GG, as SGD with momentum does (Jordan et al. use Nesterov-style momentum).
  2. Orthogonalize. Replace GG by an approximation of UV⊤UV^\top, where G=UΣV⊤G = U\Sigma V^\top is the thin singular value decomposition of GG. For full-rank GG, the ideal result keeps the singular vectors of GG and sets every singular value to 1; the finite iteration below only approximates this.
G=UΣV⊤,O=UV⊤,W←W−η OG = U \Sigma V^\top, \qquad O = U V^\top, \qquad W \leftarrow W - \eta\, O

Why this can help:

  • UV⊤UV^\top is the semi-orthogonal matrix nearest to GG. Jordan et al. observe that momentum updates for transformer weight matrices are nearly low-rank: a few directions dominate. Orthogonalizing gives the rarer directions a step of the same size.
  • Every singular value of OO is 1, so the step changes the layer's output for any input xx by at most η∥x∥\eta \lVert x \rVert, since ∥ηOx∥≤η∥x∥\lVert \eta O x \rVert \le \eta \lVert x \rVert. Bernstein & Newhouse (2024) show that UV⊤UV^\top is the direction of steepest descent when the size of a step is measured by this spectral norm.

Newton–Schulz Instead of an SVD

Computing an SVD of every weight matrix at every step would be slow on GPUs. Muon instead divides GG by its Frobenius norm, so that every singular value is at most 1, and then applies five steps of a Newton–Schulz iteration:

X←aX+b (XX⊤)X+c (XX⊤)2XX \leftarrow aX + b\,(XX^\top)X + c\,(XX^\top)^2 X

Each step uses only matrix products. It leaves the singular vectors unchanged and maps every singular value σ\sigma to aσ+bσ3+cσ5a\sigma + b\sigma^3 + c\sigma^5. Jordan et al. choose a=3.4445a = 3.4445, b=−4.7750b = -4.7750, c=2.0315c = 2.0315 so that small singular values grow quickly. The polynomial targets a band of about [0.7,1.3][0.7, 1.3], but five steps do not guarantee that every singular value reaches it: sufficiently small ones remain below it, and zero stays zero. Jordan et al. report that five steps suffice in their training experiments. They estimate the extra cost at under 1% of the training FLOPs in typical language-model training.

A Worked Example

Take the 2×22 \times 2 update

G=(110.1−0.1).G = \begin{pmatrix} 1 & 1 \\ 0.1 & -0.1 \end{pmatrix}.

Its rows are orthogonal, so its singular values are the row lengths, 2≈1.41\sqrt{2} \approx 1.41 and 0.12≈0.140.1\sqrt{2} \approx 0.14: this update moves the layer's first output ten times as much as its second. Its orthogonalized form is

UV⊤=12(111−1),UV^\top = \frac{1}{\sqrt{2}} \begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix},

which keeps the directions of the two rows, (1,1)(1, 1) and (1,−1)(1, -1), but gives both length 1. Five steps of Muon's Newton–Schulz iteration give approximately

(0.500.500.50−0.50),\begin{pmatrix} 0.50 & 0.50 \\ 0.50 & -0.50 \end{pmatrix},

with singular values about 0.71 and 0.70. The ten-to-one imbalance is gone, although the scale is not exactly 1. The same calculation in Python:

import numpy as np

def newton_schulz(G, steps=5):
    a, b, c = 3.4445, -4.7750, 2.0315
    norm = np.linalg.norm(G)
    if norm == 0:
        return np.zeros_like(G)
    X = G / norm  # Frobenius norm: every singular value is now at most 1
    for _ in range(steps):
        A = X @ X.T
        X = a * X + (b * A + c * A @ A) @ X
    return X

G = np.array([[1.0, 1.0], [0.1, -0.1]])
U, S, Vt = np.linalg.svd(G)
print(S.round(3))                                   # [1.414 0.141]
print((U @ Vt).round(3))                            # [[0.707, 0.707], [0.707, -0.707]]
O = newton_schulz(G)
print(O.round(2))                                   # [[0.5, 0.5], [0.5, -0.5]]
print(np.linalg.svd(O, compute_uv=False).round(2))  # [0.71 0.7 ]

Muon vs AdamW

AdamW

  • Scales each entry by its own running gradient statistics
  • Decoupled weight decay
  • Ignores the matrix structure of WW

Muon

  • Momentum, then orthogonalization of the whole update matrix
  • Singular values of the step are at most about 1.2η1.2\eta, so the step's spectral norm is controlled; very small ones can stay below η\eta
  • Uses the fact that WW is a linear map acting on activations, not just a list of numbers

Published Results

  • Jordan et al. (2024) report new records on two training-speed benchmarks: training a small GPT-style model on FineWeb to a validation loss of 3.28 about 1.35 times faster, and training a CIFAR-10 classifier to 94% accuracy in 2.6 instead of 3.3 A100-seconds.
  • Liu et al. (2025) found that Muon needs weight decay and a per-parameter adjustment of the update scale to work well for larger models. With both, they report about twice the compute efficiency of AdamW in compute-optimal training, and they trained Moonlight, a mixture-of-experts model with 16B parameters (3B active per token), on 5.7T tokens.
  • Wen et al. (2025) compared optimizers against carefully tuned AdamW baselines and found that the speedup of matrix-based optimizers such as Muon shrinks with model size, from about 1.4× at 0.1B parameters to 1.1× at 1.2B.

In code, the usual pattern is:

  • Muon for the 2D weight matrices of hidden layers
  • AdamW for embeddings, the output layer, normalization gains and biases

Variants

  • Muon is Scalable for LLM Training (Liu et al., 2025) adds weight decay and scales Muon's updates to the typical size of AdamW's, so that Muon can reuse the learning rate and weight decay tuned for AdamW.
  • NorMuon (Li et al., 2025) adds a per-neuron normalization after orthogonalization, because Muon's updates can give some neurons much larger steps than others.
  • MuonAll (Page et al., 2025) turns the remaining parameters into two-dimensional matrices so that Muon can handle all of them, and reports fine-tuning results on par with AdamW for models of up to about half a billion parameters.

When to Consider Muon

Muon is worth trying when:

  • You train transformer-style models at moderate to large scale
  • You're comfortable using two optimizers, Muon and AdamW, for different parameter groups
  • You want to compare against a carefully tuned AdamW baseline, since reported gains depend on how well the baseline is tuned (Wen et al., 2025)

For small models or quick experiments, AdamW remains a reasonable default.

Sources