Mathematical Foundations

Core Training⭐ 231 paper▶ Demo

2⊗

Attention

Scaled Dot-Product Attention & Transformer Layers

Optimization🔗 42 papers▶ Demo

3∇

Adam

Adam & Adaptive Gradient Methods

Optimization⭐ 112 papers▶ Demo

4⌇

Sharpness

Loss Landscapes, Sharpness & Flat Minima

Optimization🔗 42 papers▶ Demo

5∪

Double Descent

Overparameterization & Generalization, Double Descent

6Θ

NTK

Neural Tangent Kernel & Infinite-Width Limits

Theory🔗 41 paper▶ Demo

Generative Models🔗 51 paper▶ Demo

7ℤ

VAEs

Variational Autoencoders & Variational Inference

Generative Models🔗 22 papers▶ Demo

8⚔

GANs

GANs & Adversarial Divergence Minimization

Generative Models⭐ 114 papers▶ Demo

9∂

Diffusion

Diffusion, Score-Based Models & Flow Matching

Representations⭐ 141 paper▶ Demo

10◎

Embeddings

Representation Learning & Embedding Geometry

Representations🔗 32 papers▶ Demo

11⊕

Superposition

Superposition, Sparse Features & Monosemanticity

Representations🔗 52 papers▶ Demo

12⚲

Probing

Probing, Linear Classifier Probes & Activation Analysis

Representations🔗 72 papers▶ Demo

13⊛

Circuits

Transformer Circuits, Induction Heads & Mechanistic Interpretability

Scaling & Alignment🔗 73 papers▶ Demo

14↗

Scaling

Scaling Laws & Emergent Abilities

Scaling & Alignment⭐ 173 papers▶ Demo

15⚖

RLHF

Preference-Based Alignment: RLHF, Reward Modeling, Constitutional AI

Efficiency🔗 93 papers▶ Demo

16⚡

Efficiency

Efficiency: Quantization, Distillation, LoRA & Sparse MoE

17∀

Theory

Theoretical Foundations: PAC Learning, MDL & Information Bottleneck

Theory🔗 93 papers▶ Demo

Efficiency⭐ 113 papers▶ Demo

19⚙

Efficient Attention

Efficient Attention at Scale: KV Cache, GQA & FlashAttention

Representations🔗 33 papers▶ Demo

18↻

RoPE

Rotary Position Embeddings (RoPE)

Efficiency🔗 63 papers▶ Demo

20⏩

Speculative Decoding

Speculative Decoding: Lossless Multi-Token Generation

Efficiency🔗 73 papers▶ Demo

21⚡

LLM Serving

LLM Serving at Scale: Prefill, Decode & Continuous Batching

Efficiency🔗 83 papers▶ Demo

22🔀

MoE

Sparse Mixture of Experts: Routing, Load Balancing & Expert Parallelism

Efficiency🔗 43 papers▶ Demo

23⚡

MoE Serving

MoE Serving & Scheduling: Token Dispatch, All-to-All, Disaggregated Parallelism

Scaling & Alignment🔗 53 papers▶ Demo

24🎯

DPO

Direct Preference Optimization: RL-Free Alignment from Human Preferences

Scaling & Alignment🔗 33 papers▶ Demo

25👍

KTO

KTO: Alignment from Binary Feedback via Human-Aware Losses

Scaling & Alignment🔗 33 papers▶ Demo

26⚠️

Reward Hacking

Reward Hacking & Overoptimization: Goodhart's Law in Preference Optimization

Representations🔗 63 papers▶ Demo

27🔍

Sparse Autoencoders

Sparse Autoencoders at Scale: Feature Dictionaries for Mechanistic Interpretability

Representations🔗 43 papers▶ Demo

28🔬

Circuit Discovery

Automated Circuit Discovery: Patching, Attribution & Decomposition at Scale

Representations🔗 43 papers▶ Demo

29🎚️

Activation Steering

Activation Steering: Feature-Guided Interventions for Inference-Time Control

Efficiency⭐ 103 papers▶ Demo

30📏

Long Context

Long Context Engineering: RoPE Scaling, KV Compression & Memory Optimization

Core Training🔗 53 papers▶ Demo

31🔀

SSMs & Hybrids

State Space Models & Hybrid Architectures: Mamba-2, Jamba, Griffin

Representations🔗 53 papers▶ Demo

32🖼️

Multimodal VLP

Multimodal Foundations: Vision Encoders, Contrastive Learning & Cross-Attention Fusion

Representations🔗 43 papers▶ Demo

33🔤

Tokens

Tokenization & Vocabulary Design

Core Training🔗 53 papers▶ Demo

34🎲

Decoding

Decoding & Sampling: Temperature, Top-p & Inference-Time Control

35⟲

Backprop

Backpropagation & Automatic Differentiation

Optimization🔗 42 papers

Generative Models🔗 52 papers

36∇p

Score Matching

Score Matching & Score-Based Generative Models

Representations🔗 52 papers

37📝

ICL

In-Context Learning: Learning Without Weight Updates

38⚖️

OT/Wasserstein

Optimal Transport & Wasserstein Distance

Theory🔗 32 papers

Generative Models🔗 32 papers

39🌊

Flows

Normalizing Flows: Exact Likelihood via Invertible Transforms

Scaling & Alignment🔗 11 paper

40🎯

PPO

PPO: Proximal Policy Optimization

41⊕

Residuals

Residual Connections & Skip Connections

Core Training🔗 21 paper

42🎚️

CFG

Classifier-Free Guidance in Diffusion

Representations🔗 31 paper

43🔍

RAG

Retrieval-Augmented Generation (RAG)

44⚔️

Adversarial

Adversarial Examples & Robustness

Theory🔗 31 paper

45💡

Grokking

Grokking: Delayed Generalization

Theory🔗 31 paper

Representations🔗 41 paper

46🔬

Logit Lens

Logit Lens: Probing Intermediate Representations

47📉

LR Schedules

Learning Rate Schedules: Warmup, Decay & Cycling

Optimization🔗 21 paper

48🎲

Init

Weight Initialization: Xavier, He & µP

Optimization🔗 32 papers

Representations🔗 22 papers

49🔗

Contrastive

Contrastive Learning & InfoNCE

50🌐

Distributed

Distributed Training: Data, Tensor & Pipeline Parallelism

Efficiency🔗 32 papers

51🌳

Beam Search

Beam Search & Structured Decoding

Core Training🔗 21 paper

52💧

Dropout

Dropout: Stochastic Regularization

Optimization🔗 21 paper

53⚡

EBMs

Energy-Based Models & Score Functions

54📏

LayerNorm

Layer Normalization & RMSNorm

Core Training🔗 22 papers

55ℐ

Fisher Info

Fisher Information & Information Geometry

Theory🔗 42 papers

56🧭

Natural Grad

Natural Gradient & Riemannian Optimization

Optimization🔗 32 papers

57🏃

SGD+Momentum

SGD & Momentum: The Workhorses of Optimization

58⚖️

AdamW

Weight Decay & AdamW: Decoupled Regularization

59✂️

Grad Clip

Gradient Clipping & Explosion Prevention

60🎯

Label Smooth

Label Smoothing & Soft Targets

61📊

BatchNorm

Batch Normalization

Core Training🔗 12 papers

62🧪

Distillation

Knowledge Distillation: Learning from Teachers

Efficiency🔗 62 papers

63🔢

Quantization

Quantization: Compressing Models to Integers

64✂️

Pruning

Pruning: Removing Unnecessary Weights

Representations🔗 32 papers

65🔄

SSL

Self-Supervised Learning: Labels from Structure

66🌡️

Calibration

Calibration & Temperature Scaling

Theory🔗 32 papers

Core Training🔗 12 papers▶ Demo

67🚪

SwiGLU

SwiGLU & Gated Activations

68⚡

FlashAttn

FlashAttention: IO-Aware Attention

Scaling & Alignment🔗 62 papers

69📜

Constitutional

Constitutional AI: Principles-Based Alignment

70🪞

Bregman

Bregman Divergence & Mirror Descent

71🎭

RKHS

Reproducing Kernel Hilbert Spaces

72🕳️

TDA

Persistent Homology & Topological Data Analysis

73🔀

Lie Groups

Lie Groups & Equivariant Networks

Scaling & Alignment🔗 52 papers

74🧠

Test-Time

Test-Time Compute & Inference Scaling

Scaling & Alignment🔗 42 papers

75💭

CoT

Chain-of-Thought Prompting

76🌍

World Models

World Models & Model-Based RL

Scaling & Alignment🔗 22 papers

77🏭

Synth Data

Synthetic Data & Self-Improvement

Generative Models🔗 22 papers

78🎯

Consistency

Consistency Models: One-Step Diffusion

79💾

Checkpointing

Activation Checkpointing & Memory Efficiency

Efficiency🔗 21 paper▶ Demo

80👥

GQA

Grouped Query Attention (GQA)

81📋

PRMs

Process Reward Models

82🤖

RLAIF

RLAIF: AI Feedback

83🌊

Flow Match

Flow Matching & Rectified Flows

84📝

Instruct

Instruction Tuning

85🎯

Deliberative

Deliberative Alignment

Scaling & Alignment🔗 11 paper

86⚔️

Debate

AI Safety via Debate

Scaling & Alignment🔗 31 paper

87🔄

IDA

Iterated Amplification

88💪

Weak→Strong

Weak-to-Strong Generalization

89🔴

Auto RedTeam

Automated Red Teaming

90🔮

Mesa-Opt

Mesa-Optimization & Inner Alignment

91😴

Sleepers

Sleeper Agents & Alignment Faking

92⚖️

LLM-as-Judge

Model-Graded Evaluations

93🔍

Elicitation

Capability Elicitation & ELK

94🥪

Sandwich

Sandwiching Evaluations

95📊

MoD

Mixture-of-Depths

Efficiency🔗 21 paper

96🌲

MCTS-LLM

Tree Search over Thoughts

97🎬

VideoWM

Video World Models

98🔄

Self-Improve

Self-Improvement & Distillation Loops

99📉

Collapse

Model Collapse & Synthetic Data

Theory🔗 21 paper

100♾️

InfCtx

Infinite Context Architectures

Efficiency🔗 21 paper