BSNotes.in — Notes for IIT Madras BS StudentsSearch your notes⌘ K
REVISION AND PRACTICE

Week 2 Quick Revision

3 min read
On this page 5 headings

Swipe long formulas, tables or code sideways to see the rest.

Start here All of Week 2 on one page: sigmoid neuron, the learning setup, gradient descent, and the universal approximation idea.

The story in 5 lines

  1. The perceptron’s step is harsh (0.49 → no, 0.51 → yes) and not differentiable.
  2. The sigmoid (logistic) neuron σ(z)=11+e−z\sigma (z) = \frac{1}{1 + e^{-z}} is smooth, differentiable, and outputs a probability in (0, 1).
  3. Supervised learning == data + model + parameters + learning algorithm + loss. Guessing parameters doesn’t scale.
  4. Gradient descent: move opposite to the gradient (β=180∘)(\beta = 180{}^\circ ) in small steps η\eta .
  5. Universal approximation: one hidden layer of sigmoid neurons (enough of them) can approximate any continuous function to any precision — towers built from pairs of steep sigmoids.

Key formulas

Idea Formula
Sigmoid σ(z)=11+e−z,σ(0)=0.5,σ(−z)=1−σ(z)\sigma (z) = \frac{1}{1 + e^{-z}}, \sigma (0) = 0.5, \sigma (-z) = 1 - \sigma (z)
Model f(x)=11+e−(wx+b)f(x) = \frac{1}{1 + e^{-(wx + b)}}
Squared-error loss L(w,b)=(1/N)∑i(yi−f(xi))2L(w, b) = (1/N) \sum _{i} (y_{i} - f(x_{i}))^{2} (lecture code uses 12∑\frac{1}{2}\sum )
Taylor (first order) L(θ+ηu)≈L(θ)+ηuT∇θL(θ)L(\theta + \eta u) \approx L(\theta ) + \eta u^{T}\nabla \theta L(\theta )
Best direction u=−∇L(cos⁡β=−1,β=180∘)u = -\nabla L (\cos \beta = -1, \beta = 180{}^\circ )
Update w←w−η∇w,b←b−η∇bw ← w - \eta \nabla w, b ← b - \eta \nabla b
Gradients (one point, ½ squared error) ∇w=(f−y)f(1−f)x;∇b=(f−y)f(1−f)\nabla w = (f - y) f (1 - f) x; \nabla b = (f - y) f (1 - f)
Step from a sigmoid large w; step at x=−b/wx = -b/w
Tower σ(w(x−a))−σ(w(x−c))\sigma (w(x - a)) - \sigma (w(x - c))
Neurons per tower 1-D: 2; 2-D: 4 (+1 to close); n-D: 2n (+1)

Numbers worth remembering

  • Two-point example (0.5, 0.2), (2.5, 0.9): best w≈1.79,b≈−2.28w \approx 1.79, b \approx -2.28, loss 0.
  • Guess w=3,b=−1→w = 3, b = -1 \to loss ≈0.0941\approx 0.0941.
  • GD from (−2,−2),η=1(-2, -2), \eta = 1: very slow at first (flat region), loss ≈0.0177\approx 0.0177 after 100 steps, ≈0\approx 0 after 1000.

Common traps

  • Don’t forget the minus sign in the update: a negative gradient makes the weight increase.
  • ∇b\nabla b has no x factor; ∇w\nabla w has the x factor.
  • Sum the gradient over all points for (batch) gradient descent.
  • The theorem is about approximation (∣g−f∣<ϵ)(|g - f| < \epsilon ), not exact representation, and says nothing about how small the network is or how easy it is to train.
  • Perceptron network → Boolean functions exactly; sigmoid network → continuous functions approximately.

Exam words: sigmoid, logistic, differentiable, supervised learning, model, parameters, loss, error surface, gradient descent, learning rate, Taylor series, universal approximation, tower function, Cybenko, Hornik.

Quick check

  1. What does a very large w do to σ(wx+b)\sigma (wx + b)?
It makes the S-curve almost vertical — practically a step function at x=−b/wx = -b/w.
  1. f=0.8,y=0.2,x=1.∇w?f = 0.8, y = 0.2, x = 1. \nabla w?
(0.8−0.2)×0.8×0.2×1=0.6×0.16=0.096. (0.8 - 0.2) \times 0.8 \times 0.2 \times 1 = 0.6 \times 0.16 = 0.096.
  1. Name the five parts of the supervised ML setup.
Data, model, parameters, learning algorithm, loss (objective/error) function.
  1. Why is plotting the error surface not a practical learning method?
Real networks have millions of parameters; trying all combinations is intractable, and each parameter ranges over all real numbers.
  1. How many sigmoid neurons make one 1-D tower?
Two (one step up, one step down), combined by a linear output with weights +1 and −1.
YOUR STUDY PROGRESS

Finished this section? Save your progress.

Back to top ↑