Dot Product

The dot product measures alignment: it connects angles, lengths, and projections, and underlies cosine similarity in ML.

Introductory · Undergraduate mathematics · about 12 minutes

Reading map and next steps

Intuition

The idea in plain words, before the symbols.

The dot product answers the question: “How much does one vector point in the direction of another?” Its value depends on both vectors' lengths and their relative direction.

If two nonzero arrows point in the same direction, the dot product is positive and equals the product of their lengths.
If they’re perpendicular, it’s zero.
If they point in opposite directions, it’s negative.

In ML, dot products serve as similarity scores and compare queries with keys in attention. Cosine similarity instead divides the dot product by the product of the vectors' lengths; it is defined only when both vectors are nonzero.

Study prompts for this section: Intuition

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Mathematics

Definitions, assumptions and the derivation.

Equation 1 Equation 2 Equation 3

Let $u,v \in \mathbb{R}^n$ , with coordinates

u = (u_1,\ldots,u_n), \qquad v = (v_1,\ldots,v_n).

The dot product is the coordinate-wise multiply-and-sum operation

u \cdot v = \sum_{i=1}^n u_i v_i.

This definition is algebraic, but it is designed to preserve geometry. The squared length of a vector is

\|u\|^2 = u \cdot u = \sum_{i=1}^n u_i^2,

so $\|u\| = \sqrt{u\cdot u}$ . If $\theta$ is the angle between two nonzero vectors, the same operation satisfies

u \cdot v = \|u\|\,\|v\|\cos\theta.

You can read this as:

\frac{u \cdot v}{\|u\|\,\|v\|} = \cos\theta.

That ratio is cosine similarity. It removes the lengths and keeps only direction, which is why embeddings often use it as a normalized similarity score.

That immediately gives:

Orthogonality: $u \perp v \iff u\cdot v = 0$ .
Alignment: $u\cdot v > 0$ means the angle is acute, while $u\cdot v < 0$ means the angle is obtuse.
Scaling: $(au)\cdot v = a(u\cdot v)$ , so making one vector twice as long doubles the score.
Projection: the component of $u$ along $v$ is

\operatorname{proj}_v(u) = \frac{u\cdot v}{v\cdot v} \, v \quad (v \neq 0).

The scalar $\frac{u\cdot v}{v\cdot v}$ says how many copies of $v$ fit inside the part of $u$ that points along $v$ . The leftover

u_\perp = u - \operatorname{proj}_v(u)

is perpendicular to $v$ , because

u_\perp \cdot v = \left(u - \frac{u\cdot v}{v\cdot v}v\right)\cdot v = u\cdot v - \frac{u\cdot v}{v\cdot v}(v\cdot v) = 0.

In dot-product attention, this same operation appears as $q\cdot k$ : the score depends on both vector lengths and their relative direction. With nonzero lengths held fixed, a smaller angle gives a larger score; when key lengths differ, the most aligned key need not receive the largest score.

Study prompts for this section: Mathematics

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import numpy as np u = np.array([2.0, 1.0]) v = np.array([-1.0, 2.0]) dot = float(u @ v) nu =...python

import numpy as np

u = np.array([2.0, 1.0])
v = np.array([-1.0, 2.0])

dot = float(u @ v)
nu = float(np.linalg.norm(u))
nv = float(np.linalg.norm(v))
cos = dot / (nu * nv)

proj_u_on_v = (dot / float(v @ v)) * v

print("u·v =", dot)
print("cos(theta) =", cos)
print("proj_v(u) =", proj_u_on_v)
print("perp component =", u - proj_u_on_v)

Study prompts for this section: Code

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

StudyExplore the worked calculation, then choose optional practice when you are ready.Keep the same inputs while you inspect what changes and what stays fixed.

Row 0 (A) · unmasked projection.

One query · three sources · one continuous example

A dot product becomes attention

Drag q and watch three steps happen at once: match (how far q reaches along each key), share (softmax splits the attention) and mix (the output blends the values). You can explore before answering anything; go at your own pace.

Given vectors for one unmasked head, not learned word meanings.

q = [2, 1] · Compare the query with each key.

1Match: how far does q reach along each key?

Drag anywhere in the plane, or focus q and use the arrow keys.

2Share: softmax turns scores into one pie

Score q·k / √2Share

A014%

B1.41458%

C0.70728%

Softmax exponentiates each score, then divides by the total. Only differences between scores matter.

3Mix: the output is the shares’ blend of the values

Coloured arrows add w·v for each source, tip to tail. The output can never leave the triangle. The view fits the values with equal units on both axes.

q lines up most with B’s key (score 1.414), so B takes 58% of the attention; C gets 28% and A gets 14%. Output o = [-0.004, 0.868].

Give source B more than 90% of the attention.

Watch Match → Share → Mix

A 31-second guided explanation of the starting A/B/C example. This is fixed, silent playback with captions; it does not follow or reset your experiment.

Read the same calculation

Match: query (2, 1) meets keys A = (−1, 2), B = (1, 0), C = (0, 1). Their dot products are [0, 2, 1]. Divide by √2, the square root of the key width, to get [0, 1.414, 0.707].
Share: softmax subtracts the largest score, exponentiates and divides by the sum. The weights, in A/B/C order, are [0.14, 0.576, 0.284]. They sum to one; a larger match receives more weight.
Mix: multiply values A = (2, 0), B = (0, 2), C = (−1, −1) by those weights, then add their contributions. The output is [-0.004, 0.868]. The values carry the content; the query and keys determine its weighting.

Displayed numbers are rounded. This hand-authored, unmasked example explains one attention calculation; it is not a trace from a trained model.

Show every number, the formula and the Python

Change the query · one unmasked rowQuery q, x: 2Query q, y: 1

Shared normalization: m = 1.414; Z = 1.736.

Scaled scores · s = q·k/√2

−4.504.5

Attention shares w in A, B, C order · the full strip is 1

ABCSolid dot or edge: current q. Hollow ring or dashed line: same current keys and values with q = [2, 1].

Source Aindex 0
Dot
0
Scaled
0
Share
0.14
Carry w_Av_A
[0.28, 0]
Source Bindex 1
Dot
2
Scaled
1.414
Share
0.576
Carry w_Bv_B
[0, 1.152]
Source Cindex 2
Dot
1
Scaled
0.707
Share
0.284
Carry w_Cv_C
[-0.284, -0.284]

Output o = [-0.004, 0.868]

Compared with q = [2, 1]:

Scores same; shares same; output same.

Worked example, not a test or model run. Given vectors, not learned meanings. Match the query with keys; mix values, not keys. All three sources are allowed in this view.

1 · Match and scaleq · kᵢ → sᵢ = (q · kᵢ) / √2sum(qj * kj for qj, kj in zip(q, k)) / sqrt(2)

2 · Share one denominatorwᵢ = exp(sᵢ − m) / Zexps = [exp(s - max(scores)) for s in scores]weights = [e / sum(exps) for e in exps]

3 · Carry the valuescᵢ = wᵢvᵢ; o = Σᵢ cᵢsum(w * v[j] for w, v in zip(weights, values))

m is the shared maximum scaled score; Z = exp(s_A − m) + exp(s_B − m) + exp(s_C − m). Subtracting the same maximum avoids large exponentials without changing the shares. Each carry is a weighted source value; sum the three contributions coordinate by coordinate for output o.

The reference uses the same current keys and values, with q = [2, 1]. Not every query edit changes the mixture. Solid amber borders mark numerical changes; dashed teal borders mark values unchanged within 1e-12. Labels round to three decimals, not the calculation.

Source A · index 0: k_A = [-1, 2]; v_A = [2, 0]
Source B · index 1: k_B = [1, 0]; v_B = [0, 2]
Source C · index 2: k_C = [0, 1]; v_C = [-1, -1]

Inspect precise numbers and input coordinates

Finite JavaScript numbers before display rounding; −0 is kept where supplied. Tiny allowed shares can underflow to zero; that is not a mask.

q = [2, 1]
A: key=[-1, 2]; value=[2, 0]; dot=0; score=0; share=0.14002924504337802; contribution=[0.28005849008675604, 0]
B: key=[1, 0]; value=[0, 2]; dot=2; score=1.414213562373095; share=0.575975345215362; contribution=[0, 1.151950690430724]
C: key=[0, 1]; value=[-1, -1]; dot=1; score=0.7071067811865475; share=0.28399540974126003; contribution=[-0.28399540974126003, -0.28399540974126003]
m=1.414213562373095; stable denominator=1.736185425829454
output=[-0.00393691965450399, 0.8679552806894639]

Read the same calculation in Python

This runnable Python uses the current controls and the same labels. No NumPy or random inputs are needed.

Float operands and left-to-right sums from 0.0 follow browser accumulation, not exact arithmetic. Cancellation can hide a nonzero exact change; exp/sqrt implementations and display rounding can differ across runtimes.

from math import exp, sqrt

labels = ["A","B","C"]
q = [2.0, 1.0]
keys = [[-1.0, 2.0], [1.0, 0.0], [0.0, 1.0]]
values = [[2.0, 0.0], [0.0, 2.0], [-1.0, -1.0]]

# Float operands and left-to-right sums from 0.0 follow browser accumulation.
# Cancellation can hide a nonzero exact change. This is not exact arithmetic;
# exp/sqrt implementations and display rounding can differ across runtimes.
def ordered_sum(terms):
    total = 0.0
    for term in terms:
        total += term
    return total

d_k = len(q)  # key dimension, NOT number of sources or value width
raw_scores = [ordered_sum(qj * kj for qj, kj in zip(q, k)) for k in keys]
scores = [s / sqrt(d_k) for s in raw_scores]
exps = [exp(s - max(scores)) for s in scores]
total = ordered_sum(exps)
weights = [e / total for e in exps]
output = [ordered_sum(v[j] * w for w, v in zip(weights, values))
          for j in range(len(values[0]))]

for label, raw, score, weight in zip(labels, raw_scores, scores, weights):
    print(label, round(raw, 3), round(score, 3), round(weight, 3))
print("output", [round(x, 3) for x in output])

Browser reference output: [-0.004, 0.868] (rounded to three decimals).

Inspect key and value geometry

Key space only. Each grid interval is 1 coordinate unit on both axes. Amber marks the query q you control; ink marks the three fixed keys.

Value space only. Each grid interval is 1 coordinate unit on both axes. Teal marks the mixed output o; the triangle bounds its possible positions.

Step 1 of 4 · Vectors

What does this query have in common with each key?

A, B, and C name the same three sources throughout. Compare q with their keys in key space. Their values are payloads in a different space, not arrows to compare with q.

These are optional inspection controls. The connected scores, shares and mixture stay visible in the explorable above.

Read the current numbers

Shapes: q and each k have d_k = 2 coordinates. Each v and the output have d_v = 2 coordinates. Equal widths here do not make the spaces identical.

q = [q₁, q₂]; k_A = [−1, 2]; k_B = [1, 0]; k_C = [0, 1]

i names a source; j runs over A, B, C. s is a scaled score, w its share, and o the output. Displayed decimals are rounded; calculations are not.

Keys in key space
Source	Key k	q
A	[-1, 2]	[2, 1]
B	[1, 0]	[2, 1]
C	[0, 1]	[2, 1]

Reset example restores the query and first value’s x coordinate only. All other held inputs stay unchanged.

All sources are allowed here; this is not a masked sequence output. Query edits return to row 0; value edits are shared. Other rows and the held mask stay unchanged.

Practice, help and export

A study detour counts as consultation for the attempt you leave. Clearing a response or resetting values keeps its help history.

Held example · 3 queries · 3 sources. Started from the A/B/C sequence.

Study history for this visit

These record which help you used, not completion or learning scores. Each attempt keeps its own feedback and help. Reloading the page clears this history.

Guided attention notebook · worked study · sequence-abc-v1

The figure opens on the place this operation is used next: attention. One query vector $q$ is compared with three key vectors, and each score is the dot product $q\cdot k$ divided by $\sqrt 2$ , because the vectors have two coordinates. Drag $q$ and watch the first panel: the scores follow the lengths and angles described above. The other two panels, softmax and the weighted mix of values, are explained on the attention page.

For an exercise on this page's own mathematics, open Practice, help and export below the figure and choose Advanced demonstrations. Drag $u$ and $v$ , predict whether the signed projection of $u$ onto $v$ points with $v$ , nearly vanishes, or points against $v$ , then reveal it and compare the result with the signs of $u\cdot v$ and $\cos\theta$ .

Study prompts for this section: Demo

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Focus on

In focus

Concept: Dot Product

What is the smallest example of Dot Product you can work through by hand, and what does it show?

BeforeVector SpacesNextDerivatives

DetailsLinear Algebra

About this itemLinear Algebra

2 sources citedChecking this browser

Reference id

concept:linear-algebra/dot-product

Math3 equationsu = (u_1,\ldots,u_n), \qquad v = (v_1,\ldots,v_n).CodeDot Product code example 1u = np.array([2.0, 1.0])DemoDot Product interactive demoChange one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.

What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

The dot product measures alignment: it connects angles, lengths, and projections, and underlies cosine similarity in ML.

Choose what to look for (no demo yet)01 / Intuition

Illustration of two vectors, their angle, and a projection shadow.

What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Dot Product.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

The dot product measures alignment: it connects angles, lengths, and projections, and underlies cosine similarity in ML.

4 of 4 sections writtenNo live demo yet

Visual cueWhich part of the picture should you look at first?

Highlight size2/4

Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Dot Product.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.

book · 2020Mathematics for Machine LearningDeisenroth, Faisal, and Ong

Why it is cited

Grounds inner products, vector geometry, and the linear algebra notation reused across attention and optimization.

Open source

book · 2023Introduction to Linear Algebra (6th ed.)Strang

Why it is cited

Wellesley-Cambridge Press. Chapter 1 introduces the dot product, vector length and the angle formula cos θ = u·v / (‖u‖ ‖v‖).

Open source

What each claim rests on1 claim, each with its sources, where it appears on this page, and its caveats.

Claims and sources

The dot product measures alignment: it connects angles, lengths, and projections, and underlies cosine similarity in ML.

Checked against the cited passages

The checks were made by this site, not by an independent reviewer.

For vectors u,v in R^n, the dot product sum_i u_i v_i is a coordinate multiply-sum that also encodes geometry: u dot v = ||u||||v||cos(theta), so sign and size track alignment, cosine similarity removes lengths, and proj_v(u) scales v by (u dot v)/(v dot v).

Sources

Mathematics for Machine LearningGrounds inner products, vector geometry, and the linear algebra notation reused across attention and optimization.

What the sources support

MML introduces inner products as algebraic operations with induced norms, angles, orthogonality, and projections; the page's formulas and demo show those relationships in 2D and connect normalized dot produc...

Where to see it on this page

Mathematics for Machine Learning Equation 2 Code example 1 Demo

Equation 2

u \cdot v = \sum_{i=1}^n u_i v_i.

Caveat

Applies to finite-dimensional real vectors with v != 0, and to angle/cosine only when both vectors are nonzero. Dot-product magnitude still includes vector lengths; cosine normalizes direction only. Excludes...

StatusChecked against the cited passages.

MML supports the claim: it defines the R^n dot product as x^T y=sum_i x_i y_i, uses inner products to define norms, angles and orthogonality, defines cos omega=<x,y>/(||x|| ||y||), and derives projection onto span(b) as (<x,b>/<b,b>)b. With the dot product this is proj_v(u)=(u dot v)/(v dot v)v, which the page's equations, code and demo compute.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Dot Product

Try the idea in your own words

The dot product measures alignment: it connects angles, lengths, and projections, and underlies cosine similarity in ML.

Concept · Selected for practice

Dot Product

What it rests on: Sources: Mathematics for Machine Learning; Introduction to Linear Algebra (6th ed.)

Context and links

Linear Algebra

Choose a task

Explain the mechanism

For Dot Product: What is the smallest example of Dot Product you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

Your answerNo answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

Where am I stuck? (optional)

Your own description, not an automatic diagnosis

Choose one, or leave this unspecified. Select it again to clear it.

Take your draft to a feedback conversation

No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

Write an attempt before copying a feedback prompt.

A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.

Ask about this pageClose

ConceptDot ProductSources: Mathematics for Machine Learning; Introduction to Linear Algebra (6th ed.)u = np.array([2.0, 1.0])

Your question

Choose what your question is about

Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptLinear Algebra

Dot Product

Starting question

What is the smallest example of Dot Product you can work through by hand, and what does it show?

SourcesCheck the 2 cited sources listed under Sources.Notes can be saved for this item

Points of view on this item

Fixed prompts generated from this item. They are not comments from people or independent reviews.

Learner’s first stepAsk what would make "Dot Product" feel predictable rather than familiar.

Assumption

The 2 cited sources must support this exact item, not just the surrounding topic.

Source-checking summary

Connect the definition to one equation, piece of code or demo before widening the discussion.

Proposed experiment

Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

Next action

You can state the mechanism in your own words

Evidence4 checks

ObservationChecking for a saved observation

ActionReady for one action

PromptLearner prompt ready to copy

Go to this item

01ObservationChecking what this browser has saved

02EvidenceChecking for a saved observation

03SourcesCheck the 2 cited sources listed under Sources.

04Next stepSave one next action

Local action draftNo local draft saved yetOpen when you are ready to write one next action

Local action draft

This draft stays in this browser and is attached to this item only.

Draft noteNext action

No local draft saved.

Evidence to inspect

What the 2 cited sources say about this exact item
The definition, its prerequisites, and a contrasting concept
The equation or code that makes the concept concrete
One demo state that shows what stays the same as the inputs change

What would resolve this

You can state the mechanism in your own words
You can name the prerequisite that would clear up the confusion
You can predict how the result changes when one input changes

Reference id

concept/concept-notebook/linear-algebra/dot-product
concept:linear-algebra/dot-product

Return to reading

Reading map

In this concept

Change one control at a time and name what responds.

Try the demo

Continue the curriculum

Part I · unit 2 of 13

Derivatives

Connections and sources

Domain

Linear Algebra

linear-algebra
geometry
similarity

Prerequisites

Vector Spaces

Leads to

NormsNo page yet
Linear TransformationsNo page yet
Scaled Dot-Product Attention & Transformer Layers

Linear IndependenceNo page yet
Basis and SpanNo page yet

Study aid

Ask an assistant about this notebook

Compose a question with this notebook as context, then copy it into an AI assistant you already use.

Compose a question

Your question

GoalComfortStyleStuck on

Copy-only tool: this panel composes a prompt from this page and your choices above, for an AI assistant you already use. Nothing you enter here is sent from this page.