BSNotes.in — Notes for IIT Madras BS StudentsSearch your notes⌘ K
SECTION 04 / 4

Choosing the Learning Rate and Momentum, and Line Search

5 min read
On this page 7 headings
BUILD INTUITION WITH NUMBERS

Try three learning rates on the same gradient

Again use L(w) = ½w², current w = 2 and gradient 2. Compare candidate steps from the same starting point.

  1. A small rate improves the loss.

    η=0.1:wnew=1.8,L=1.62\eta=0.1:\quad w_{new}=1.8,\quad L=1.62

  2. This candidate reaches the minimum for this particular quadratic.

    η=1:wnew=0,L=0\eta=1:\quad w_{new}=0,\quad L=0

  3. Too large a rate overshoots and worsens the loss.

    η=3:wnew=−4,L=8\eta=3:\quad w_{new}=-4,\quad L=8

What changed? A trial-based line search compares candidate losses. Rate 1 wins here; that does not make it a universally good learning rate.

Quick check: Would you choose η = 3 for this step?

No. It raises the loss from 2 to 8.

Swipe long formulas, tables or code sideways to see the rest.

Start here The learning rate η\eta decides how big every step is. Too small and training crawls; too big and it jumps around wildly. This page gives practical tips to pick η\eta , ways to shrink it over time (annealing), a schedule for momentum, and line search — trying several η\eta at every step.

1. Why not just use a big learning rate?

Idea: gentle slopes give tiny gradients, so multiply them by a large η\eta to make the steps bigger.

Lecture example — η=10\eta = 10 On gentle regions this helps. But on steep regions, the already large gradient is blown up even more. The parameters jump far past the minimum and the loss jumps around.

What we’d like is a learning rate that adjusts itself to the gradient — big where the slope is gentle, small where it is steep. Such adaptive algorithms come next in the course (AdaGrad, RMSProp, Adam).

Pick a sensible starting η, then let it shrink as training goes on.
Pick a sensible starting η, then let it shrink as training goes on.
Open image ↗
Too-small learning rate crawls, too-large jumps over the valley, and three schedules shrink eta over time

2. Tips for the initial learning rate

  1. Tune on a log scale: try 0.0001, 0.001, 0.01, 0.1, 1.0.
  2. Run a few epochs with each; find the one that works best.
  3. Do a finer search around it. If 0.1 was best, now try 0.05, 0.2, 0.3.

Disclaimer from the lecture: these are heuristics — there is no clear winning strategy.

3. Tips for annealing (decaying) the learning rate

Start with a reasonable η\eta and reduce it as training goes on.

Step decay

  • Halve the learning rate after every 5 epochs, or
  • Halve it after an epoch if the validation error is higher than it was at the end of the previous epoch.

Exponential decay η=η0⋅e−kt\eta = \eta _{0} \cdot e^{-kt}, where η0\eta _{0} and k are hyperparameters and t is the step number.

(The slide prints this as “η=η0−kt\eta = \eta _{0}^{-kt}”; the standard exponential decay formula is η0⋅e−kt\eta _{0} \cdot e^{-kt}.)

1/t decay η=η0/(1+kt)\eta = \eta _{0} / (1 + kt), where η0\eta _{0} and k are hyperparameters and t is the step number.

Worked numbers(η0=0.1,k=0.1) (\eta _{0} = 0.1, k = 0.1)

t Exponential η0e−kt\eta _{0}e^{-kt} 1/t decay η0/(1+kt)\eta _{0}/(1 + kt) Step decay (halve every 5)
0 0.1 0.1 0.1
1 0.1×e−0.1=0.09050.1 \times e^{-0.1} = 0.0905 0.1/1.1=0.09090.1/1.1 = 0.0909 0.1
5 0.1×e−0.5=0.06070.1 \times e^{-0.5} = 0.0607 0.1/1.5=0.06670.1/1.5 = 0.0667 0.05
10 0.1×e−1=0.03680.1 \times e^{-1} = 0.0368 0.1/2=0.050.1/2 = 0.05 0.025

4. Tips for momentum

A schedule suggested by Sutskever et al., 2013:

βt=min⁡(1−2−1−log2(⌊t/250⌋+1),βmax)\beta _t = \min (1 - 2^{-1 - log_{2}(⌊t/250⌋ + 1)}, \beta _{\mathrm{max}})

where βmax\beta _{\mathrm{max}} is chosen from {0.999, 0.995, 0.99, 0.9, 0}.

What it does: β\beta starts at 0.5 and grows every 250 steps:

t ⌊t/250⌋ + 1 βt\beta _t (before the min)
0–249 1 1−2−1=0.51 - 2^{-1} = 0.5
250–499 2 1−2−2=0.751 - 2^{-2} = 0.75
500–749 3 1−2−1−log23=1−1/6=0.8331 - 2^{-1-log_{2}3} = 1 - 1/6 = 0.833
750–999 4 1−2−3=0.8751 - 2^{-3} = 0.875
1000–1249 5 1−1/10=0.91 - 1/10 = 0.9

So momentum is small at the start (careful) and larger later (confident), capped at βmax\beta _{\mathrm{max}}.

5. Line search

In practice a line search is often done to find a better η\eta at each step.

def do_line_search_gradient_descent(max_epochs):
    w, b, etas = -2, -2, [0.1, 0.5, 1, 2, 10]
    for i in range(max_epochs):
        dw, db = 0, 0
        for x, y in zip(X, Y):
            dw += grad_w(w, b, x, y)
            db += grad_b(w, b, x, y)
        min_error = 10000                    # some large value
        for eta in etas:                     # try every candidate
            temp_w = w - eta * dw
            temp_b = b - eta * db
            if error(temp_w, temp_b) < min_error:
                best_w, best_b = temp_w, temp_b
                min_error = error(best_w, best_b)
        w, b = best_w, best_b                # keep the best one
Compute the gradient once, try every η, keep the step with the lowest loss.
Compute the gradient once, try every η, keep the step with the lowest loss.
Open image ↗
Several candidate learning rates are tried from the same point; the one giving the lowest loss is kept
  • Update w using different values of η\eta .
  • Keep the updated value that gives the lowest loss.
  • So at each step we use the best η\eta from the available choices.

Worked numbers (first step from w=b=−2w = b = -2, lecture data; checked in code) Loss after the step: η=0.1→0.41581,η=0.5→0.41577,η=1→0.41573,η=2→0.41564,η=10→\eta = 0.1 \to 0.41581, \eta = 0.5 \to 0.41577, \eta = 1 \to 0.41573, \eta = 2 \to 0.41564, \eta = 10 \to 0.41488. Line search keeps η=10\eta = 10 here, because we start on a very flat region.

Plus: convergence is faster than vanilla gradient descent with a fixed η\eta . Minus: many more computations in each step (one loss evaluation per candidate η\eta ).

Lecture example — line search in action Blue: gradient descent with fixed η=1\eta = 1. Black: gradient descent with line search. Line search converges faster. It shows some oscillations too, but these are different from momentum/NAG oscillations — they come from picking abigηa big \eta when it happens to give the lowest loss for that one step.

In one line: Tune η\eta on a log scale, shrink it over time (step, exponential or 1/t decay), grow momentum over time, or use line search to pick the best η\eta at every step at extra cost.

Exam words: learning rate, log-scale search, annealing, step decay, exponential decay, 1/t decay, hyperparameter, validation error, Sutskever momentum schedule, βmax\beta _{\mathrm{max}}, line search.

Quick check

  1. η0=0.5\eta _{0} = 0.5, step decay halving every 5 epochs. η\eta at epoch 12?
Halved twice (after 5 and 10): 0.5 → 0.25 → 0.125.
  1. 1/t decay with η0=1,k=0.5\eta _{0} = 1, k = 0.5. η\eta at t=4t = 4?
11+0.5×4=1/3≈0.333. \frac{1}{1 + 0.5 \times 4} = 1/3 \approx 0.333.
  1. Exponential decay η0=0.2,k=0.1\eta _{0} = 0.2, k = 0.1. η\eta at t=10t = 10?
0.2×e−1=0.2×0.368=0.0736. 0.2 \times e^{-1} = 0.2 \times 0.368 = 0.0736.
  1. Sutskever schedule with βmax=0.99\beta _{\mathrm{max}} = 0.99. β\beta at t=300t = 300?
⌊300/250⌋+1=2→1−2−2=0.75.min⁡(0.75,0.99)=0.75. ⌊300/250⌋ + 1 = 2 \to 1 - 2^{-2} = 0.75. \min (0.75, 0.99) = 0.75.
  1. Line search with 5 candidate η\eta values: how many loss evaluations per step?
5 (one per candidate), plus one gradient computation.
YOUR STUDY PROGRESS

Finished this section? Save your progress.

Back to top ↑