BSNotes.in — Notes for IIT Madras BS StudentsSearch your notes⌘ K
REVISION AND PRACTICE

Week 3 Quick Revision

3 min read
On this page 5 headings

Swipe long formulas, tables or code sideways to see the rest.

Start here All of Week 3 on one page: feedforward networks, output/loss choices, and backpropagation.

The story in 5 lines

  1. A feedforward network computes ak=bk+Wkhk−1a_{k} = b_{k} + W_{k}h_{k-1} and hk=g(ak)h_{k} = g(a_{k}) for each hidden layer, then y^=O(aL)\hat{y} = O(a_L).
  2. Parameters θ=\theta = all WkW_{k} and bkb_{k}; learned by gradient descent, θ\theta ← θ−η∇θ\theta - \eta \nabla \theta .
  3. Real-valued outputs → linear output + squared error. Classes → softmax + cross-entropy (L=−log⁡y^L = -\log \hat{y}_ℓ).
  4. Backpropagation passes blame from the loss down through the layers using the chain rule.
  5. With softmax + cross-entropy, the output gradient is just y^−e(ℓ)\hat{y} - e(\ell ).

Key formulas

Idea Formula
Pre-activation ak=bk+Wkhk−1(h0=x)a_{k} = b_{k} + W_{k} h_{k-1} (h_{0} = x)
Activation hk=g(ak);y^=O(aL)h_{k} = g(a_{k}); \hat{y} = O(a_L)
Softmax y^j=eaLj/∑ieaLi\hat{y}_{j} = e^{a_{\mathrm{Lj}}} / \sum _{i} e^{a_{\mathrm{Li}}}
Cross-entropy L=−∑cyclog⁡y^c=−log⁡y^ℓL = -\sum _c y_c \log \hat{y}_c = -\log \hat{y}_\ell
Gradient w.r.t. y^\hat{y} ∇y^L=−(1/y^ℓ)e(ℓ)\nabla \hat{y} L = -(1/\hat{y}_\ell ) e(\ell )
Output gradient ∇aLL=−(e(ℓ)−y^)\nabla a_L L = -(e(\ell ) - \hat{y})
Hidden units ∇hk−1L=WkT∇akL\nabla h_{k-1} L = W_{k}^{T} \nabla a_{k} L
Through activation ∇ak−1L=∇hk−1L⊙g′(ak−1)\nabla a_{k-1} L = \nabla h_{k-1} L \odot g'(a_{k-1})
Weights ∇WkL=∇akL⋅hk−1T\nabla W_{k} L = \nabla a_{k} L \cdot h_{k-1}^{T}
Biases ∇bkL=∇akL\nabla b_{k} L = \nabla a_{k} L
Sigmoid derivative σ′=σ(1−σ)\sigma ' = \sigma (1 - \sigma )
tanh derivative tanh⁡′=1−tanh⁡2\tanh ' = 1 - \tanh ^{2}

Shapes

Thing Shape
WiW_{i} (hidden, n units each) n×nn \times n
WLW_L (k outputs) k×nk \times n
bi/bLb_{i} / b_L n / k
∇Wk\nabla W_{k} same as WkW_{k}

Common traps

  • ∇y^L\nabla \hat{y} L (only one non-zero entry, −1/y^-1/\hat{y}_ℓ) is not the same as ∇aLL\nabla a_L L (ŷ − e(ℓ), all entries non-zero).
  • Use WTW^{T} (transpose) when passing gradients down.
  • ⊙ is element-wise, not a matrix product.
  • Sigmoid output can’t predict values outside (0, 1) → use linear for real values.
  • The entries of y^−e(ℓ)\hat{y} - e(\ell ) always add to 0.
  • Slide erratum: one slide writes ∇WL∈Rn×k\nabla W_L \in \mathbb{R}^{n\times k}; it should be k×nk \times n, the same as WLW_L.

Exam words: feedforward network, pre-activation, activation, output function, linear output, softmax, cross-entropy, log-likelihood, backpropagation, chain rule, one-hot vector, outer product, element-wise product, forward/backward propagation.

Quick check

  1. What does softmax guarantee about its outputs?
All are positive and they add up to 1 — a probability distribution.
  1. y^=[0.5,0.3,0.2]\hat{y} = [0.5, 0.3, 0.2], true class 1. Loss and ∇aL\nabla a_L?
L=−ln⁡0.5=0.693.∇aL=[−0.5,0.3,0.2]. L = -\ln 0.5 = 0.693. \nabla a_L = [-0.5, 0.3, 0.2].
  1. Write ∇Wk\nabla W_k in terms of ∇ak\nabla a_k and hk−1h_{k-1}.
∇Wk=∇akhk−1T\nabla W_k = \nabla a_k h_{k-1}^{T} (outer product).
  1. A sigmoid hidden unit has output 0.3. What is g′?
0.3×0.7=0.21. 0.3 \times 0.7 = 0.21.
  1. Order the steps: update, backward pass, forward pass.
Forward pass → backward pass → update.
YOUR STUDY PROGRESS

Finished this section? Save your progress.

Back to top ↑