Backpropagation — the Idea, and Gradients at the Output Layer

Start here The network made a mistake. Who is to blame? Backpropagation answers this by passing the blame backwards, layer by layer, using the chain rule. This page builds the intuition and then derives the first and most famous result: at a softmax + cross-entropy output, the gradient is simply $\hat{y} -$ (true label).

To learn $W_{112}$ (one weight in the first layer) with gradient descent we need $\partial L(\theta )/\partial W_{112}$. How does a weight deep inside the network affect the loss at the top?

Imagine a network with one neuron per layer$: x_{1} \to a_{11} \to h_{11} \to a_{21} \to h_{21} \to a_L1 \to \hat{y} \to L(\theta ).$

The chain rule gives the answer as one long product:

$\partial L/\partial W_{111} = (\partial L/\partial \hat{y}) \cdot (\partial \hat{y}/\partial a_L1) \cdot (\partial a_L1/\partial h_{21}) \cdot (\partial h_{21}/\partial a_{21}) \cdot (\partial a_{21}/\partial h_{11}) \cdot (\partial h_{11}/\partial a_{11}) \cdot (\partial a_{11}/\partial W_{111})$

Compress it — each layer can reuse what the layer above already worked out:

• $\partial L/\partial W_{111} = (\partial L/\partial h_{11}) \cdot (\partial h_{11}/\partial W_{111})$
• $\partial L/\partial W_{211} = (\partial L/\partial h_{21}) \cdot (\partial h_{21}/\partial W_{211})$
• $\partial L/\partial W_L11 = (\partial L/\partial a_L1) \cdot (\partial a_L1/\partial W_L11)$

Everyday example In a relay race, the time lost by the first runner reaches the finish line through every later runner. To find the first runner’s share, you go backwards from the finish, one runner at a time.

1. We get a loss $-\log \hat{y}$_ℓ at the output. Who is responsible?
2. We tell the output layer: “You are not producing the right output — take responsibility.”
3. The output layer says: “I take my share, but I am only as good as the hidden layer and weights below me” — after all, $\hat{y} = O(W_L h_{L-1} + b_L)$.
4. We ask $W_L, b_L$ and $h_L$ what’s wrong. $W_L$ and $b_L$ take full responsibility. $h_L$ says: “I’m only as good as my pre-activation layer.”
5. The pre-activation says: “I’m only as good as the hidden layer and weights below me.”
6. Continue down… the responsibility lies with all the weights and biases.

But instead of talking to each weight directly, it is easier to talk to them through the hidden and output layers — and that is exactly what the chain rule does:

$\partial L/\partial W_{111} =$$\partial L/\partial \hat{y} \cdot \partial \hat{y}/\partial a_{3}$ (talk to the output layer) · $\partial a_{3}/\partial h_{2} \cdot \partial h_{2}/\partial a_{2}$ (previous hidden layer) · $\partial a_{2}/\partial h_{1} \cdot \partial h_{1}/\partial a_{1}$ (previous hidden layer) · $\partial a_{1}/\partial W_{111}$ (now talk to the weight)

Roadmap (with softmax output + cross-entropy loss):

1. Gradient w.r.t. output units — this page.
2. Gradient w.r.t. hidden units — next page.
3. Gradient w.r.t. weights and biases — next page.

Loss: $L(\theta ) = -\log \hat{y}$_$\ell$ (ℓ = true class).

$\partial L/\partial \hat{y}_{i} = \partial (-\log \hat{y}_\ell )/\partial \hat{y}_{i} =$$-1/\hat{y}$_ℓ if $i = \ell$, and 0 otherwise (the loss only contains $\hat{y}$_ℓ).

Compactly: $\partial L/\partial \hat{y}_{i} = -$𝟙$(i = \ell ) / \hat{y}$_ℓ, where 𝟙$(i = \ell )$ is 1 if $i = \ell$, else 0.

As a vector:

$\nabla \hat{y} L(\theta ) = -(1/\hat{y}_\ell ) \cdot e(\ell )$

where e(ℓ) is a k-dimensional vector with 1 at position ℓ and 0 everywhere else (a one-hot vector).

What we really want is $\partial L/\partial a_{\mathrm{Li}}$. Does $\hat{y}$_ℓ depend on $a_{\mathrm{Li}}$? Yes — through softmax, every $\hat{y}$ depends on every $a_L$:

$\hat{y}_\ell = e^{a_L\ell } / \sum _{i}' e^{a_{\mathrm{Li}}'}$

Derivation:

$\partial (-\log \hat{y}_\ell )/\partial a_{\mathrm{Li}} = -(1/\hat{y}_\ell ) \cdot \partial \hat{y}_\ell /\partial a_{\mathrm{Li}}$

Use the quotient rule $\partial /\partial x [g/h] = (\partial g/\partial x)/h - g(\partial h/\partial x)/h^{2}$, with $g = e^{a_L\ell }$ and $h = \sum _{i}' e^{a_{\mathrm{Li}}'}$:

• $\partial g/\partial a_{\mathrm{Li}} = 𝟙(\ell = i) \cdot e^{a_L\ell }$
• $\partial h/\partial a_{\mathrm{Li}} = e^{a_{\mathrm{Li}}}$

So

$\partial \hat{y}_\ell /\partial a_{\mathrm{Li}} = 𝟙(\ell = i) \cdot e^{a_L\ell }/\sum - (e^{a_L\ell }/\sum ) \cdot (e^{a_{\mathrm{Li}}}/\sum ) = 𝟙(\ell = i) \cdot \hat{y}_\ell - \hat{y}_\ell \cdot \hat{y}_{i}$

Put it back:

$\partial L/\partial a_{\mathrm{Li}} = -(1/\hat{y}_\ell ) \cdot (𝟙(\ell = i) \hat{y}_\ell - \hat{y}_\ell \hat{y}_{i}) =$$-(𝟙(\ell = i) - \hat{y}_{i})$

As a vector:

$\nabla a_L L(\theta ) = -(e(\ell ) - \hat{y}) = \hat{y} - e(\ell )$

Beautiful and simple: prediction minus the one-hot truth.

Lecture example — with numbers $\hat{y} = [0.659, 0.242, 0.099]$, true class $=$ apple $(\ell = 1)$, so $e(\ell ) = [1, 0, 0]$. $\nabla a_L L = \hat{y} - e(\ell ) = [0.659 - 1, 0.242 - 0, 0.099 - 0] =$ [−0.341, 0.242, 0.099]. Meaning: increase the apple score (negative gradient → gradient descent pushes it up), decrease the others.

Note: the entries of $\nabla a_L$ always add up to 0 ($\sum \hat{y}_{i} = 1$ and $\sum e(\ell )_{i} = 1$).

In one line: Backpropagation passes blame from the loss down through the layers with the chain rule; at a softmax + cross-entropy output the gradient w.r.t. the pre-activation is just $\hat{y} - e(\ell )$.

Exam words: backpropagation, chain rule, deep thin network, gradient w.r.t. output units, one-hot vector e(ℓ), indicator 𝟙$(i = \ell )$, quotient rule, $\nabla a_L L = -(e(\ell ) - \hat{y})$.