How to Solve Week 4 Questions

Question: A network is trained on 5000 data points using mini-batch gradient descent with batch size 50, for 100 epochs. How many times does each parameter get updated?

Question: Mini-batch gradient descent is run on a dataset of size 1000 with batch size 25 for 10 epochs. Find the number of times each parameter is updated.

Question: You classify 200 bird species from 100,000 audio clips, starting with vanilla gradient descent. Q18: How many parameter updates per epoch? Q19: You switch to mini-batch GD with $B = 50$. How many updates per epoch now?

Question: 12 training samples, vanilla gradient descent for 4 epochs. Learning rate $\eta _{t} = 0.8 \exp (-0.2t)$, where t starts at 0 and increases only when the parameters are updated. What is $\eta _{t}$ at the end of training? (3 decimals)

Question: Minimise $L(w) = (w + 3)^{2}$ with momentum: $u_{t} = \beta u_{t-1} - \nabla w_{t}, w_{t} = w_{t-1} + \eta u_{t}$. $\eta = 0.2, \beta = 0.8, w_{0} = 5, u_{0} = 0$. Find $w_{2}$ (2 decimals).

Question: Linear regression $\hat{y} = wx$ with $J(w) = \frac{1}{2m} \sum (wx_{i} - y_{i})^{2}$, mini-batch size $m = 2, \eta = 0.1, \beta = 0.9, w_{0} = 1, u_{-1} = 0$. Rules: $u_{t} = \beta u_{t-1} + \nabla J(w_{t}), w_{t+1} = w_{t} - \eta u_{t}$. First mini-batch: $(x, y) = (1, 2)$ and (2, 4). $Q14: \nabla J(w_{0})$ for this batch? (−2.5 · −5.0 · −7.5 · 5.0) Q15: the velocity $u_{0}$ after the first iteration (1 decimal)?

Question: Momentum GD with $\beta = 0.8$ and $\eta = 0.1$ optimises f(w) of two parameters. At time t the history vector is $u_{t-1} = (1, -1)$. After one iteration, $w_{t+1} = w_{t}$. Q66: Find $\nabla w_{t}$. ((−0.8, 0.8) · (0.08, −0.08) · (−0.08, 0.08) · (1, −1)) Q67: the angle between $u_{t-1}$ and $\nabla w_{t}$ in degrees, in [0, 360]?

Question: Minimise $f(x, y) = x^{2} + y^{2}$ from any point $(x_{0}, y_{0}) \ne$ origin. Q74: Run two steps of GD with $\eta = 0.2$ to $get (x_{2}, y_{2}) = \alpha (x_{0}, y_{0})$. Find α (2 decimals). Q75: Momentum with $u_{-1} = (0, 0), \eta = 0.2: u_{t} = \beta u_{t-1} + \nabla f(x_{t}, y_{t}), (x_{t+1}, y_{t+1}) = (x_{t}, y_{t}) - \eta u_{t}$. What $\beta$ makes two steps reach the global minimum, i.e. $(x_{2}, y_{2}) = (0, 0)$? (2 decimals)

Question: The graph shows training loss across epochs for learning rates 0.001, 0.01 and 1.0. Graph A: smooth but very slow convergence. Graph B: fast convergence to a low loss. Graph C: oscillates and does not converge. Match each graph to its learning rate.

Question: Two runs of gradient descent on $L(w) = w^{2}$ from the same start $w_{0} = 2$ give Graph X and Graph Y with two learning rates. Which could be correct? (X uses $\eta = 1.1, Y$ uses $\eta = 0.3$) · (X uses $\eta = 0.3, Y$ uses $\eta = 1.1$)

Question: $L(w) = (w - 5)^{2}$ is minimised from $w_{0} = -5$ with $\eta = 0.04$ using vanilla $GD (\beta = 0)$ and momentum GD with $\beta \in \{0.5, 0.9, 0.99\}$. Match each curve to its $\beta$.

Question: Which statement correctly distinguishes the Momentum optimiser from Nesterov Accelerated Gradient (NAG)?

1. Momentum uses the current gradient to update the velocity, while NAG uses a look-ahead gradient based on a tentative future position. (b) Momentum is computationally faster because it skips the look-ahead gradient. (c) NAG adjusts the step size dynamically. (d) Momentum uses an adaptive learning rate, NAG a fixed one.

Question: For gradient descent $w_{t+1} = w_{t} - \eta \nabla w_{t}$, select the correct behaviours: (a) update is tiny on a gentle loss surface (b) tiny on a steep surface (c) large on a steep surface (d) large on a gentle surface.

Question: Training on billions of data points, batch gradient descent is computationally infeasible. What is the main reason, requiring an alternative like SGD?

Question: Which statements about training neural networks are TRUE?

1. $SGD (B = 1)$ computes the true gradient of the total loss at each step. (b) Mini-batch GD with batch size B on N points does ⌈N/B⌉ updates per epoch. (c) Momentum GD can overshoot a narrow valley because accumulated velocity carries the parameters past the minimum. (d) In vanilla (batch) GD, each step is guaranteed to decrease the loss for a small enough learning rate on a smooth loss.