Computation Graphs

A computation graph breaks a calculation into nodes so values flow forward and sensitivities flow backward.

Introductory · Undergraduate mathematics · about 13 minutes

Reading map and next steps

Chain Rule has no page here yet. Read OpenStax, Calculus Volume 1, Section 3.6: The Chain Rule.

Ready? What is the derivative of sin(3x)?

Show the answer

3 cos(3x): the derivative of the outer function, cos(3x), times the derivative of the inner one, 3.

Intuition

The idea in plain words, before the symbols.

Suppose one intermediate value gets used twice. If $a=xy$ , $b=\sin(a)$ , and $c=a+b$ , then $a$ affects $c$ directly and also through $b$ . How should the derivative remember both uses without differentiating one huge expanded formula by hand?

A computation graph makes that bookkeeping visible.

Instead of treating a function as one large expression, we split it into small operations: multiply here, apply a nonlinearity, add there, reuse an intermediate value. Each operation becomes a node, and each dependency becomes an edge.

A computation graph is not just an expression tree. The expanded expression $xy+\sin(xy)$ repeats $xy$ . The graph stores $a=xy$ once and records that two later operations depend on that same value.

This matters because the graph gives backpropagation a route. Values move forward through the graph to produce an output or loss. Derivatives move backward through the same graph, using local slopes on each edge. The chain rule stops feeling like one giant symbolic derivative and becomes bookkeeping over small local changes.

Study prompts for this section: Intuition

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Mathematics

Definitions, assumptions and the derivation.

Equation 1 Equation 2 Equation 3

In this first example, every node is scalar. Let $x,y\in\mathbb R$ , and draw edges from inputs to the values that depend on them. The graph for one forward pass is a directed acyclic graph: later nodes depend on earlier nodes, not the other way around.

Consider the scalar computation

a = xy,\qquad b = \sin(a),\qquad c = a + b.

The graph has inputs $x,y$ , intermediate nodes $a,b$ , and output $c$ . The forward pass stores each node value. The important detail is reuse: $a$ is stored once, but two later uses depend on it. The backward pass asks how a small change in each node would affect the output.

For any node $v$ , write

\bar v = \frac{\partial c}{\partial v}.

The output seed is $\bar c=1$ . The final add node $c=a+b$ sends one unit of sensitivity to both inputs:

\bar a \mathrel{+}= \bar c\frac{\partial c}{\partial a}\Big|_{\text{direct}} = 1,\qquad \bar b \mathrel{+}= \bar c\frac{\partial c}{\partial b} = 1.

The sine branch also sends a local contribution into the same stored node $a$ . The rule is local: a node's sensitivity $\bar v$ (also called its adjoint or cotangent) is the sum of the contributions from each node that uses it directly. The demo hides the sine-branch contribution until you predict whether adding it leaves $\bar a$ below, near, or above the direct contribution.

The backward pass does not expand every symbolic path; each downstream node has already summarized everything beyond it.

For vector or tensor nodes, the same idea uses Jacobians or local vector-Jacobian product rules. With column-vector cotangents, if $v_j=f(v_i)$ and $J_{ji}=\partial v_j/\partial v_i$ , the reverse update is

\bar v_i \mathrel{+}= J_{ji}^{\mathsf T}\bar v_j.

Reverse-mode autodiff is the automated version of this bookkeeping: save forward values, then run local backward rules in reverse topological order.

Study prompts for this section: Mathematics

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Code

The same calculation as runnable code, in the notation of the derivation.

Code example 1import math x, y = 2.0, 3.0 # Forward pass: store intermediate node values. a = x * y b = mat...python

import math

x, y = 2.0, 3.0

# Forward pass: store intermediate node values.
a = x * y
b = math.sin(a)
c = a + b

# Reverse pass bookkeeping: every downstream use appends one local contribution.
bar_c = 1.0
bar_a = 0.0
bar_b = 0.0
bar_x = 0.0
bar_y = 0.0

# c = a + b sends one unit to both inputs.
bar_a += bar_c * 1.0
bar_b += bar_c * 1.0

# b = sin(a) sends another contribution into reused node a.
bar_a += bar_b * math.cos(a)

# a = x * y sends the accumulated sensitivity to x and y.
bar_x += bar_a * y
bar_y += bar_a * x

print("c:", round(c, 4))
print("stored a once:", a)
print("downstream uses of a:", ["direct add edge", "sine edge"])
print("bar_a:", round(bar_a, 4))
print("dc/dx:", round(bar_x, 4))
print("dc/dy:", round(bar_y, 4))

The code is deliberately manual. The += updates are the graph mechanism: a reused node collects every downstream local contribution before sending its accumulated sensitivity to its own inputs.

Study prompts for this section: Code

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Demo

Change an input, predict the result, and see which quantities respond.

ExploreChange one control at a time and name what responds.An optional observation guide follows the demo.

Explore Computation Graphs

Loading the interactive demo…

Observation guide

Optional and ungraded. Opening a guide saves nothing.

Choose what to inspect in Computation Graphs, then open its guide.

Use the sliders to change $x$ and $y$ . In Forward mode, watch the stored value $a=xy$ feed both $b=\sin(a)$ and the final add node $c=a+b$ .

Before switching into the backward numbers, predict whether the hidden accumulated sensitivity at the reused node $a$ should land lower than, nearly equal to, or higher than the direct contribution from $c=a+b$ .

Case A and Case B are two starting points whose names give nothing away. Choose an answer first, then reveal how the downstream paths combine before $a$ sends sensitivity back to $x$ and $y$ .

Study prompts for this section: Demo

Study prompts

Each button copies a prompt about this section to your clipboard, for an AI assistant you already use. Nothing is sent by this site.

Focus on

In focus

Concept: Computation Graphs

What is the smallest example of Computation Graphs you can work through by hand, and what does it show?

BeforeDerivativesNextReverse-Mode Automatic Differentiation

DetailsCalculus

About this itemCalculus

2 sources citedChecking this browser

Reference id

concept:calculus/computation-graphs

Math3 equationsa = xy,\qquad b = \sin(a),\qquad c = a + b.CodeComputation Graphs code example 1print("downstream uses of a:", ["direct add edge", "sine edge"])DemoComputation Graphs interactive demoChange one control at a time and name what responds.

After the first pass

Go deeper

Optional material for a second reading. Sources are always shown; the other panels open from their headings or from these links, and closing one keeps what you wrote or revealed.

What to look for in each sectionChoose a section and what you expect to see in it, then read short advice.

Section by section

Choose what to look for in each section

A computation graph breaks a calculation into nodes so values flow forward and sensitivities flow backward.

Choose what to look for (no demo yet)01 / Intuition

Illustration of computation graph nodes with forward value flow and backward sensitivity arrows.

What to look for

Start with the picture, metaphor, or geometric mechanism.

Choose first

Choose what you expect to see in this section of Computation Graphs.

Questions to ask of the pictureChoose a part of the picture and what you expect, then read short advice.

Looking at the picture

Questions to ask of the picture

A computation graph breaks a calculation into nodes so values flow forward and sensitivities flow backward.

4 of 4 sections writtenNo live demo yet

Visual cueWhich part of the picture should you look at first?

Highlight size2/4

Question

Which part of the picture should you look at first?

Choose first

Pick the part of the picture you expect to explain Computation Graphs.

Sources

References for this page

A reference shows where an idea comes from; it does not vouch for every step on this page.

paper · 2018Automatic differentiation in machine learning: a surveyBaydin et al.

Why it is cited

Journal of Machine Learning Research 18(153), 1–43. Grounds computation graphs as the program structure used by automatic differentiation systems.

Open source

book · 2016Deep LearningGoodfellow, Bengio, and Courville

Why it is cited

MIT Press. Section 6.5.1 introduces computational graphs; Section 6.5 runs the chain rule backwards over them, adding the contributions from every node that uses a value.

Open source

What each claim rests on1 claim, each with its sources, where it appears on this page, and its caveats.

Claims and sources

A computation graph breaks a calculation into nodes so values flow forward and sensitivities flow backward.

Checked against the cited passages

The checks were made by this site, not by an independent reviewer.

For one executed differentiable program, a computation graph uses elementary-operation nodes and dependency edges, stores each reused intermediate node value once in the forward pass, and makes reverse-mode/backprop local: downstream sensitivities from every use accumulate at a node before being sent to its inputs.

Sources

Automatic differentiation in machine learning: a surveyJournal of Machine Learning Research 18(153), 1–43. Grounds computation graphs as the program structure used by automatic differentiation systems.

What the sources support

Baydin et al. describe AD over evaluation traces of elementary operations, computational graphs for dependency relations, and reverse mode as a forward pass that populates intermediate variables/records grap...

Where to see it on this page

Automatic differentiation in machine learning: a survey Equation 1 Equation 2 Code example 1 Demo

Equation 1

a = xy,\qquad b = \sin(a),\qquad c = a + b.

Equation 2

\bar v = \frac{\partial c}{\partial v}.

Caveat

Covers one run of a program built from differentiable operations, drawn as a graph without cycles, and the local backward rule for scalar and vector nodes; not symbolic simplification, in-place updates, chec...

StatusChecked against the cited passages.

Baydin et al. support the mechanism for one run of a program: it becomes a trace of elementary operations and a graph of dependencies between intermediates; reverse mode runs forward, stores intermediates and records dependencies, then propagates adjoints backward, and a reused variable's adjoint sums its downstream contributions before input derivatives are formed. The page's equations, code and demo use a=xy, b=sin(a), c=a+b, store a once, add its direct and sine-path cotangents and pass the sum to x and y.

Explain it without the pageWrite your own explanation before asking for feedback. Your draft and any hints stay while this page is open.

Practice · Computation Graphs

Try the idea in your own words

A computation graph breaks a calculation into nodes so values flow forward and sensitivities flow backward.

Concept · Selected for practice

Computation Graphs

What it rests on: Sources: Automatic differentiation in machine learning: a survey; Deep Learning

Context and links

Calculus

Choose a task

Explain the mechanism

For Computation Graphs: What is the smallest example of Computation Graphs you can work through by hand, and what does it show? Explain your answer, including what changes, why, and which assumption matters.

Your answerNo answer yet

A rough first thought is enough. Your draft stays when you change tasks.

Your draft stays on this page and is cleared when you leave or reload.

A little help · Explain

Open one hint at a time. These are suggestions, not your answer or a grade.

0 of 3 hints shown for this question.

Clearing your answer keeps this help record. Outside help is your own report; this page cannot check it.

Where am I stuck? (optional)

Your own description, not an automatic diagnosis

Choose one, or leave this unspecified. Select it again to clear it.

Take your draft to a feedback conversation

No AI feedback runs here. You can copy a prompt to use elsewhere; nothing is sent automatically. Review the text before sharing, and leave out private information.

Write an attempt before copying a feedback prompt.

A draft, or an AI reply to it, is not a test of what you have learned. To check that, try a different case later without help.

Ask about this pagePick one item on this page and copy a prompt about it into an AI assistant you already use. Notes you write here stay in this browser.

Ask about this pageClose

ConceptComputation GraphsSources: Automatic differentiation in machine learning: a survey; Deep Learningprint("downstream uses of a:", ["direct add edge", "sine edge"])

Your question

Choose what your question is about

Pick the idea, equation, source, code, claim, misconception or demo state you want to ask about. The prompt you copy includes it, so the answer can stay on that item.

Next local actionNo local draft saved yet

Open the draft below to save one note and next action in this browser.

conceptCalculus

Computation Graphs

Starting question

What is the smallest example of Computation Graphs you can work through by hand, and what does it show?

SourcesCheck the 2 cited sources listed under Sources.Notes can be saved for this item

Points of view on this item

Fixed prompts generated from this item. They are not comments from people or independent reviews.

Learner’s first stepAsk what would make "Computation Graphs" feel predictable rather than familiar.

Assumption

The 2 cited sources must support this exact item, not just the surrounding topic.

Source-checking summary

Connect the definition to one equation, piece of code or demo before widening the discussion.

Proposed experiment

Change one input, then check that the same relationship holds in the mathematics, the code and the demo.

Next action

You can state the mechanism in your own words

Evidence4 checks

ObservationChecking for a saved observation

ActionReady for one action

PromptLearner prompt ready to copy

Go to this item

01ObservationChecking what this browser has saved

02EvidenceChecking for a saved observation

03SourcesCheck the 2 cited sources listed under Sources.

04Next stepSave one next action

Local action draftNo local draft saved yetOpen when you are ready to write one next action

Local action draft

This draft stays in this browser and is attached to this item only.

Draft noteNext action

No local draft saved.

Evidence to inspect

What the 2 cited sources say about this exact item
The definition, its prerequisites, and a contrasting concept
The equation or code that makes the concept concrete
One demo state that shows what stays the same as the inputs change

What would resolve this

You can state the mechanism in your own words
You can name the prerequisite that would clear up the confusion
You can predict how the result changes when one input changes

Reference id

concept/concept-notebook/calculus/computation-graphs
concept:calculus/computation-graphs

Return to reading

Reading map

In this concept

Change one control at a time and name what responds.

Try the demo

Continue the curriculum

Part I · unit 4 of 13

Reverse-Mode Automatic Differentiation

Connections and sources

Domain

Calculus

calculus
autodiff
backpropagation

Prerequisites

Derivatives
Chain Rule has no page here yet. Read OpenStax, Calculus Volume 1, Section 3.6: The Chain Rule.
Ready? What is the derivative of sin(3x)?
Show the answer
3 cos(3x): the derivative of the outer function, cos(3x), times the derivative of the inner one, 3.

Leads to

Reverse-Mode Automatic Differentiation

Backpropagation

Study aid

Ask an assistant about this notebook

Compose a question with this notebook as context, then copy it into an AI assistant you already use.

Compose a question

Your question

GoalComfortStyleStuck on

Copy-only tool: this panel composes a prompt from this page and your choices above, for an AI assistant you already use. Nothing you enter here is sent from this page.