Ask what should change when the equation is manipulated, then let the visualization test that expectation.
Foundation Lab
State Space Models & Hybrid Architectures: Mamba-2, Jamba, Griffin
Long context (#30) exposes transformer's Achilles heel (quadratic attention + KV cache)—SSMs are the architectural escape hatch
#31SSMs & HybridsCore Training
key equation
h_{t+1} = Ah_t + Bx_t, \quad y_t = Ch_tSelected Foundation Object
Keep the equation fixed; move through the evidence.
Concept 31 of 100SSMs & HybridsCore Training / Phase 6: Modern efficiency & inference
Current question
SSMs can be taught as "attention with structured kernel"—both compute weighted sums over past, SSMs do it via recurrence/scan
h_{t+1} = Ah_t + Bx_t, \quad y_t = Ch_tUse the runnable panel, the key equation, and canonical papers as separate forms of evidence for the same object.
The useful learning product is the reusable mechanism you can carry into another model, paper, or engineering tradeoff.
Use prerequisites, dependents, and semantic links to repair the next gap without leaving the object behind.
Why It Matters for Modern Models
- Long context (#30) exposes transformer's Achilles heel (quadratic attention + KV cache)—SSMs are the architectural escape hatch
- Mamba-2/SSD frames Structured State-Space Duality: SSMs and attention are dual, both compute weighted sums but SSMs do it via linear recurrence
- Jamba: hybrid Transformer-Mamba + MoE for capacity, reports strong performance up to 256K tokens—shows hybrids dominate not pure SSMs
- RecurrentGemma/Griffin: mix linear recurrences with local attention for efficiency + long-sequence suitability
- Why SSMs work for language now is selectivity (input-dependent behavior), not just O(T) complexity—otherwise you get bland smoothing kernel
What Tutorials Skip
What is still poorly explained in textbooks and papers:
- SSMs can be taught as "attention with structured kernel"—both compute weighted sums over past, SSMs do it via recurrence/scan
- Reason "SSMs work for language now" is selectivity (input-dependent behavior)—otherwise you get smoothing that can't do sharp retrieval
- Hybrids exist because you want: local attention for short-range syntax + recurrence/SSM for long-range memory—neither alone is optimal
- Constant state memory is key advantage—KV cache grows with T, SSM state stays fixed size, enabling truly unbounded context
- Hardware-friendly is critical—linear recurrence maps to efficient scans/cumsum, while attention needs custom kernels (FlashAttention)
Interactive Visualization
Core Math (Optional Deep Dive)
If you want intuition first, start with the key equation and the visualization. Come back here for the full walkthrough.
Key Equation
SSMs replace global attention with recurrences/structured kernels, or mix both (local attention + recurrence) for long-context efficiency. Key: linear-time sequence modeling.
SSM recurrence:
State update is linear—constant memory, time.
Equivalent convolution/kernel view:
SSMs can be viewed as attention with structured kernel—both compute weighted sums over past.
Hybrid gating intuition (generic template):
Captures Griffin-style "recurrence + local attention" hybrids—best of both worlds.