Continuous Function

NLP & Speech

Word2Vec, GloVe, and Subword Embeddings

See how context windows, global co-occurrence counts, and character n-grams shape word vectors, and why nearest-neighbor semantics can mislead.

status: reviewimportance: importantdifficulty 3/5math: undergraduateread: 17mlive demo
Back to NLP & SpeechNext: Embedding, Unembedding, and Weight Tying (review)

Concept Structure

Word2Vec, GloVe, and Subword Embeddings

01Intuition

Start with the picture, metaphor, or geometric mechanism.

02Math

Make the objects explicit and connect them with notation.

03Code

Mirror the equations with runnable implementation details.

04Interactive Demo

Manipulate the mechanism and watch the idea respond.

4prerequisites
4next concepts
3related links

Learner Contract

What this page should let you do.

You are here becauseSee how context windows, global co-occurrence counts, and character n-grams shape word vectors, and why nearest-neighbor semantics can mislead.

This NLP & Speech concept is the current idea: keep the same invariant visible across Intuition, Math, Code, Interactive Demo.

By the end4/4 sections ready | runnable code expected | live demo

Explain the mechanism, trace the main notation, and test one prediction in the live demo.

Do this firstIntuition

Read the intuition before the notation; the math should name a mechanism you already felt.

Test the linkManipulate one control and predict the visible change.Then continue to Embedding, Unembedding, and Weight Tying (review)

Claim/source review status

Substantive review recorded

3/3 claims have bounded review metadata; still check caveats and source scope.Metadata-derived; review may be AI-assisted. Not a human certification.
Claims3/3 reviewed
Sources8 cited
Codeattached
Demolive
Reviewed2026-07-03
Updatedpage 2026-07-03

Learning item flow

4/4 sections readyAsk about thisResearch room
ConceptWord2Vec, GloVe, and Subword EmbeddingsNLP & SpeechEquationWord2Vec, GloVe, and Subword Embeddings equation 1Exact equationCodeWord2Vec, GloVe, and Subword Embeddings runnable code 1Exact runnable codeDemoWord2Vec, GloVe, and Subword Embeddings interactive demoInteractive demoClaimWord2Vec-style embeddings learn word vectors from local context predi...Exact claim checkSourceCS224N Lecture Notes Part I: Word Vectors IExact source
ConceptWord2Vec, GloVe, and Subword EmbeddingsNLP & Speech
6 sources attachedLocal snapshot ready
concept:nlp-speech/word2vec-glove-subword-embeddings
CodenearbyPredictbefore revealRoomitem handoff
01IntuitionStart with the picture, metaphor, or geometric mechanism.02MathMake the objects explicit and connect them with notation.03CodeMirror the equations with runnable implementation details.04Interactive DemoManipulate the mechanism and watch the idea respond.
01

01

Intuition

Build the mental picture first so the rest of the page has something to attach to.

Section prompt

A word embedding is not the meaning of a word written down by hand. It is a learned vector whose geometry comes from a training signal.

The distributional idea is simple: words that appear in similar contexts should become useful neighbors. If river often appears near stream, water, and flow, while money appears near loan, cash, and bank, a training objective can push the vectors into different regions of space.

Word2Vec makes this local. In skip-gram, a center word predicts words in a small context window. A positive pair such as (river, stream) should get a larger dot product; sampled negative words such as money should get a smaller dot product.

GloVe makes the global count table visible. Instead of only looking at one window at a time, it fits vector dot products to the logarithm of corpus co-occurrence counts, with a weighting function so common and rare events do not dominate in the same way.

Subword embeddings repair a different failure. A pure word lookup has no vector for a word it has never seen. fastText-style models build a word vector from character n-gram vectors, so riverbank can receive a composed vector from pieces such as riv, iver, ban, and ank.

The caveat is part of the concept: nearest neighbors are evidence, not truth. Static word vectors usually give one vector per word type, so bank has to share space between river-bank and money-bank contexts. Subword pieces help with rare forms, but spelling is not meaning.

Source spine: Stanford CS224N Word Vectors I and Word Vectors II; Jurafsky and Martin, SLP3 Chapter 6; Mikolov et al., Efficient Estimation of Word Representations and Distributed Representations of Words and Phrases; Pennington et al., GloVe; Bojanowski et al., Subword Information; D2L, The Word2Vec Dataset.

02

02

Math

Translate the story into symbols, assumptions, and a derivation you can inspect.

Section prompt
Equation 1log⁡σ(uw⊤vc)+∑k=1Klog⁡σ(−uw⊤vnk),\log \sigma(u_w^\top v_c) + \sum_{k=1}^{K}\log \sigma(-u_w^\top v_{n_k}),Equation 2J=∑i,jf(Xij)(ui⊤vj+bi+b~j−log⁡Xij)2.J = \sum_{i,j} f(X_{ij}) \left( u_i^\top v_j + b_i + \tilde{b}_j - \log X_{ij} \right)^2.

Let VV be a vocabulary and let uw∈Rdu_w \in \mathbb{R}^d be the target embedding for word ww. Let vc∈Rdv_c \in \mathbb{R}^d be the context embedding for context word cc.

For a skip-gram positive pair (w,c)(w,c) with negative samples n1,…,nKn_1,\dots,n_K, the negative-sampling objective for one training example is:

log⁡σ(uw⊤vc)+∑k=1Klog⁡σ(−uw⊤vnk),\log \sigma(u_w^\top v_c) + \sum_{k=1}^{K}\log \sigma(-u_w^\top v_{n_k}),

where σ(t)=1/(1+exp⁡(−t))\sigma(t)=1/(1+\exp(-t)).

The first term raises the dot product for the observed center-context pair. The negative terms lower dot products for sampled words that did not appear as the context in this example.

GloVe starts from a global co-occurrence matrix XX, where XijX_{ij} counts how often word jj appears in the context of word ii. It fits:

J=∑i,jf(Xij)(ui⊤vj+bi+b~j−log⁡Xij)2.J = \sum_{i,j} f(X_{ij}) \left( u_i^\top v_j + b_i + \tilde{b}_j - \log X_{ij} \right)^2.

Only nonzero co-occurrences enter this log-count form. The weighting function ff controls how strongly each count matters.

For fastText-style subword embeddings, let G(w)G(w) be the set of character n-grams for word ww. A simplified composed word vector is:

zw=1∣G(w)∣∑g∈G(w)zg.z_w = \frac{1}{|G(w)|} \sum_{g \in G(w)} z_g.

The important distinction is:

  • Word2Vec: local context prediction updates vectors.
  • GloVe: global co-occurrence counts become log-count targets.
  • Subword embeddings: character n-gram vectors compose a word vector, including for rare or unseen forms.
03

03

Code

Keep the implementation aligned with the notation so the algorithm is legible.

Section prompt
Runnable code 1import math import numpy as np river = np.array([0.20, 0.40]) stream = np.array([0.50, 0.20])...python

This small script mirrors the lab: one skip-gram update, one GloVe log-count table, and one subword composition.

import math
import numpy as np

river = np.array([0.20, 0.40])
stream = np.array([0.50, 0.20])
money = np.array([-0.40, -0.30])
lr = 0.35

def sigmoid(x): return 1 / (1 + math.exp(-x))
def cosine(a, b): return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

positive_grad = (1 - sigmoid(river @ stream)) * stream
negative_grad = -sigmoid(river @ money) * money
river_after = river + lr * (positive_grad + negative_grad)

print("cos(river, stream):", round(cosine(river, stream), 3), "->", round(cosine(river_after, stream), 3))
print("cos(river, money):", round(cosine(river, money), 3), "->", round(cosine(river_after, money), 3))

counts = np.array([[6, 5, 2, 0], [5, 4, 1, 0], [1, 0, 3, 4], [0, 0, 0, 6]])
log_targets = np.full_like(counts, np.nan, dtype=float)
log_targets[counts > 0] = np.log(counts[counts > 0])
print("GloVe log-count target river-flow:", round(log_targets[0, 1], 3))

ngrams = {
    "riv": [0.28, 0.62], "ive": [0.18, 0.55], "ver": [0.15, 0.50],
    "erb": [0.05, 0.32], "rba": [0.00, 0.20], "ban": [-0.12, 0.05], "ank": [-0.10, 0.02],
}
riverbank = np.mean(np.array(list(ngrams.values())), axis=0)
bank = np.array([0.00, 0.18])
print("subword riverbank vector:", np.round(riverbank, 3))
print("closest demo neighbors:", sorted(
    {"river": river, "stream": stream, "bank": bank, "money": money}.items(),
    key=lambda item: -cosine(riverbank, item[1])
)[:2])
04

04

Interactive Demo

Use direct manipulation to connect the explanation to a moving system.

Section prompt

Prediction check: inspect the context window, qualitative vector map, co-occurrence table, and subword pieces. Then predict which pair should move closer after a skip-gram update and which mechanism can create a vector for unseen riverbank before revealing the exact movement, log-count targets, and nearest-neighbor caveat.

The lab hides cosine values, update arrows, GloVe log-count targets, and the subword-composed vector until you commit. The goal is to connect the map to the training signal instead of treating an embedding map as magic semantics.

Live Concept Demo

Explore Word2Vec, GloVe, and Subword Embeddings

The stage is code-native and interactive. Use it to test the explanation against the mechanism.

difficulty 3/5undergraduatecode-aligned
Demo Prediction Checkpoint

Manipulate one control and predict the visible change.

Commit to what Word2Vec, GloVe, and Subword Embeddings should make visible before reading the result.

After The First Pass

Turn the concept into an inspected object.

Once the invariant is visible in the intuition, math, code, and demo, use these panels to inspect the mechanism visually, check source support, practice the idea, and attach a grounded research question.

Mechanism Storyboard

See the idea move before the page explains it

See how context windows, global co-occurrence counts, and character n-grams shape word vectors, and why nearest-neighbor semantics can mislead.

Prediction open01 / Intuition
Prediction lens

Start with the picture, metaphor, or geometric mechanism.

Commit first

Before reading further, choose the kind of change Word2Vec, GloVe, and Subword Embeddings should make visible.

Visual Inquiry

Make the image answer a mathematical question

See how context windows, global co-occurrence counts, and character n-grams shape word vectors, and why nearest-neighbor semantics can mislead.

4/4 stages readyLive demo connected
Prediction

Which visible object should carry the first intuition?

Commit first

Pick the cue that should make Word2Vec, GloVe, and Subword Embeddings easier to reason about before the page gives the answer.

Source Grounding

Canonical references for the mechanism on this page.

course-notes · 2025CS224N Lecture Notes Part I: Word Vectors IStanford CS224N

Course source for distributional semantics, co-occurrence matrices, cosine similarity, and Word2Vec skip-gram framing.

Open source
course-notes · 2025CS224N Lecture Notes Part II: Word Vectors IIStanford CS224N

Course source for negative sampling, GloVe, analogy structure, and word-vector evaluation caveats.

Open source
book · 2026Speech and Language Processing, Chapter 6: Vector Semantics and EmbeddingsJurafsky and Martin

Book source for vector semantics, distributional similarity, embeddings, Word2Vec, and GloVe context.

Open source
paper · 2013Efficient Estimation of Word Representations in Vector SpaceMikolov, Chen, Corrado, and Dean

Primary source for CBOW and skip-gram architectures for learning continuous word representations from local context.

Open source
paper · 2013Distributed Representations of Words and Phrases and their CompositionalityMikolov, Sutskever, Chen, Corrado, and Dean

Primary source for negative sampling, subsampling frequent words, phrase vectors, and analogy behavior in Word2Vec-style embeddings.

Open source
paper · 2014GloVe: Global Vectors for Word RepresentationPennington, Socher, and Manning

Primary source for the GloVe weighted least-squares objective over log global co-occurrence counts.

Open source

Claim Review

See how context windows, global co-occurrence counts, and character n-grams shape word vectors, and why nearest-neighbor semantics can mislead.

Status3 substantive reviews recorded

Claims without a substantive review badge still need exact source-support review.

Sources6 references

cs224n-2019-word-vectors-1, cs224n-2019-word-vectors-2, jurafsky-martin-slp3-vector-semantics, mikolov-2013-efficient-word-representations, mikolov-2013-distributed-representations-phrases, pennington-2014-glove

Local checks4 local checks

Use equations, runnable code, and demos to check whether the source support is operational.

Substantively reviewedWord2Vec-style embeddings learn word vectors from local context prediction: skip-gram uses a center word to predict nearby context words, while efficient training commonly uses negative sampling or related approximations instead of a full softmax over the vocabulary.Claim metadata: source checked

CS224N and Mikolov et al. support CBOW/skip-gram framing and efficient objectives; the second Mikolov paper supports negative sampling and analogy behavior; D2L supports center/context/negative-example construction for implementation.

Sources: CS224N Lecture Notes Part I: Word Vectors I, CS224N Lecture Notes Part II: Word Vectors II, Efficient Estimation of Word Representations in Vector Space, Distributed Representations of Words and Phrases and their Compositionality, d2l-word2vecThe demo uses a two-dimensional hand-computable projection and one positive/negative update; real Word2Vec trains high-dimensional target and context matrices over large corpora.A bounded review summary is present; still check caveats and exact reference scope.

Checked CS224N word-vector notes, Mikolov Word2Vec papers, and D2L Word2Vec data-preparation material for local context-prediction mechanics and negative-sampling scope.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-03
source-span-cs224n-2019-word-vectors-1source-span-cs224n-2019-word-vectors-2source-span-mikolov-2013-efficient-word-representationssource-span-mikolov-2013-distributed-representations-phrasesEquation 1Runnable code 1Interactive demo
Substantively reviewedGloVe learns word vectors by fitting dot products and biases to the logarithm of nonzero global co-occurrence counts under a weighting function, making global corpus statistics visible in the training objective.Claim metadata: source checked

Pennington et al. support the weighted log-co-occurrence objective; CS224N and SLP3 support the pedagogical connection between co-occurrence matrices, distributional similarity, and embedding geometry.

Sources: CS224N Lecture Notes Part II: Word Vectors II, Speech and Language Processing, Chapter 6: Vector Semantics and Embeddings, GloVe: Global Vectors for Word RepresentationThe demo uses a tiny count table and log-count target cells; it does not train a full GloVe model, tune the weighting function, or evaluate downstream quality.A bounded review summary is present; still check caveats and exact reference scope.

Checked the GloVe paper, CS224N Word Vectors II, and SLP3 vector semantics for global co-occurrence objective support and log-count caveats.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-03
source-span-pennington-2014-gloveEquation 2Interactive demo
Substantively reviewedSubword embedding models such as fastText represent a word using character n-gram vectors, so an unseen or rare surface form can receive a composed vector even when it lacks a learned whole-word row.Claim metadata: source checked

Bojanowski et al. support character n-gram summation and rare/unseen word handling; SLP3 supports subword embeddings in the broader vector-semantics context.

Sources: bojanowski-2017-fasttext-subword, Speech and Language Processing, Chapter 6: Vector Semantics and EmbeddingsSubword composition captures morphology and spelling evidence, not guaranteed meaning; a composed vector can be plausible and still wrong for polysemy, idioms, names, or corpus bias.A bounded review summary is present; still check caveats and exact reference scope.

Checked fastText subword paper and SLP3 vector semantics for character n-gram representation scope and caveats.

Reviewer: codex-local-primary-source-audit; reviewed 2026-07-03
source-span-bojanowski-2017-fasttext-subwordRunnable code 1Interactive demo

Source support candidates

course-notes 2025CS224N Lecture Notes Part I: Word Vectors I

Course source for distributional semantics, co-occurrence matrices, cosine similarity, and Word2Vec skip-gram framing.

course-notes 2025CS224N Lecture Notes Part II: Word Vectors II

Course source for negative sampling, GloVe, analogy structure, and word-vector evaluation caveats.

book 2026Speech and Language Processing, Chapter 6: Vector Semantics and Embeddings

Book source for vector semantics, distributional similarity, embeddings, Word2Vec, and GloVe context.

paper 2013Efficient Estimation of Word Representations in Vector Space

Primary source for CBOW and skip-gram architectures for learning continuous word representations from local context.

Mechanism checks

Equation 1
log⁡σ(uw⊤vc)+∑k=1Klog⁡σ(−uw⊤vnk),\log \sigma(u_w^\top v_c) + \sum_{k=1}^{K}\log \sigma(-u_w^\top v_{n_k}),
Equation 2
J=∑i,jf(Xij)(ui⊤vj+bi+b~j−log⁡Xij)2.J = \sum_{i,j} f(X_{ij}) \left( u_i^\top v_j + b_i + \tilde{b}_j - \log X_{ij} \right)^2.
Runnable code 1import math import numpy as np river = np.array([0.20, 0.40]) stream = np.array([0.50, 0.20])...Demo stateLive mechanism probe

Practice Loop

Try the idea before it explains itself

See how context windows, global co-occurrence counts, and character n-grams shape word vectors, and why nearest-neighbor semantics can mislead.

Readiness0/3 checks ready
Predict

Before touching the demo, predict one visible change that should happen in Word2Vec, GloVe, and Subword Embeddings.

Hint 1

Reveal when your model needs a nudge.

Hint 2

Reveal when your model needs a nudge.

Hint 3

Reveal when your model needs a nudge.

Grounded research drawerClose
ConceptWord2Vec, GloVe, and Subword EmbeddingsNLP & Speech
Runnable code comparisonWord2Vec, GloVe, and Subword Embeddings runnable code 1stream = np.array([0.50, 0.20])Prediction before revealWord2Vec, GloVe, and Subword Embeddings interactive demoManipulate one control and predict the visible change.
Grounded room questionWhat is the smallest example that makes Word2Vec, GloVe, and Subword Embeddings click without losing the math?Local snapshot ready

Research Room

Attach the question to a claim, equation, code, or demo

Pick the concept, equation, source, runnable code, claim, misconception, or demo state before asking for help. The handoff keeps that page item in context.
Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptNLP & Speech

Word2Vec, GloVe, and Subword Embeddings

Attached question

What is the smallest example that makes Word2Vec, GloVe, and Subword Embeddings click without losing the math?

Local action draftNo local draft saved yetExpand only when ready to capture one local next action
Local action draft

This draft stays in this browser, attached to the selected learning item.

No local draft saved.
Evidence to inspect
  • References to inspect: attached references on this page.
  • Definition, prerequisite, and contrast concept links
  • The equation or runnable code that makes the concept operational
  • One demo state that shows the invariant instead of a slogan
What would resolve this
  • The learner can state the mechanism in their own words
  • The learner can name the prerequisite that would repair confusion
  • The learner can predict how the mechanism changes under one perturbation
Grounded AI handoff

I am working in Continuous Function's research reading room. Object: concept - Word2Vec, GloVe, and Subword Embeddings Selected item key: recorded for copy. Context: NLP & Speech Page anchor: recorded for copy. Open question: What is the smallest example that makes Word2Vec, GloVe, and Subword Embeddings click without losing the math? Evidence to inspect: - References to inspect: attached references on this page. - Definition, prerequisite, and contrast concept links - The equation or runnable code that makes the concept operational - One demo state that shows the invariant instead of a slogan What would resolve this: - The learner can state the mechanism in their own words - The learner can name the prerequisite that would repair confusion - The learner can predict how the mechanism changes under one perturbation Answer as a careful research tutor: stay source-grounded, separate verified evidence from assumptions, name the relevant math objects, and end with one next action.

View it in context
concept/concept-notebook/nlp-speech/word2vec-glove-subword-embeddings concept:nlp-speech/word2vec-glove-subword-embeddings