BSNotes.in — Notes for IIT Madras BS StudentsSearch your notes⌘ K
SECTION 01 / 4

The Sigmoid Neuron — a Smooth Perceptron

4 min read
On this page 6 headings
BUILD INTUITION WITH NUMBERS

From a score to a soft prediction

Use weight 2, bias −1, and input 0.5.

  1. First calculate the weighted score.

    z=wx+b=2(0.5)−1=0z=wx+b=2(0.5)-1=0

  2. Pass the score through the sigmoid.

    σ(0)=11+e0=12\sigma(0)=\frac{1}{1+e^0}=\frac12

  3. A larger input produces a larger score.

    x=1⇒z=1⇒σ(1)≈0.7311x=1\Rightarrow z=1\Rightarrow\sigma(1)\approx0.7311

What changed? Unlike a hard threshold, sigmoid output changes smoothly as the score changes.

Quick check: What does the sigmoid output when its score is zero?

0.5, regardless of how the weights and inputs produced that zero score.

Swipe long formulas, tables or code sideways to see the rest.

Start here A perceptron says a hard “yes” or “no”. A movie rated 0.51 → like, 0.49 → dislike. That is too harsh. The sigmoid neuron replaces the sudden jump with a smooth S-shaped curve, and gives a probability instead of a hard 0/1. This one change is what makes learning with calculus (gradient descent) possible.

1. Where we are heading

Week 1 was about Boolean functions: inputs in {0, 1}ⁿ, output in {0, 1}.

Now we want arbitrary functions y=f(x)y = f(x), where:

  • x∈Rnx \in \mathbb{R}^{n} (real-number inputs), and
  • y∈Ry \in \mathbb{R} (real-number output).

Can a network (approximately) represent such functions? Before answering, we must move from perceptrons to sigmoid neurons.

2. The problem: the perceptron is too harsh

Recall: a perceptron fires if the weighted sum is greater than the threshold (−w0)(-w_{0}).

Lecture example — critics’ rating Decide like/dislike using only x1=x_{1} = criticsRating, a number between 0 and 1. Threshold 0.5 (so −w0=0.5-w_{0} = 0.5), weight w1=1w_{1} = 1.

  • criticsRating =0.51→0.51≥0.5→= 0.51 \to 0.51 \ge 0.5 \to like
  • criticsRating =0.49→0.49<0.5→= 0.49 \to 0.49 < 0.5 \to dislike

A tiny difference of 0.02 flips the decision completely. That seems harsh.

This is not because of this example or these numbers. It is a property of the perceptron itself: it behaves like a step function. When z=∑i=1nwixiz = \sum _{i=1}^{n} w_{i}x_{i} crosses −w0-w_{0}, the output suddenly jumps from 0 to 1.

In real life we expect a smoother decision that changes gradually from 0 to 1.

Left: the step jumps at 0.5. Right: the sigmoid changes slowly, so 0.49 and 0.51 give almost the same answer.
Left: the step jumps at 0.5. Right: the sigmoid changes slowly, so 0.49 and 0.51 give almost the same answer.
Open image ↗
Step function jumps at the threshold while the sigmoid rises smoothly

3. The sigmoid (logistic) neuron

The most common sigmoid function is the logistic function:

y=11+exp⁡(−(w0+∑i=1nwixi))y = \frac{1}{1 + \exp (-(w_{0} + \sum _{i=1}^{n} w_{i}x_{i}))}

We write σ\sigma (sigma) for it: σ(z)=11+e−z\sigma (z) = \frac{1}{1 + e^{-z}}.

Feel the numbers (z is the weighted sum including the bias):

z e−ze^{-z} σ(z)=11+e−z\sigma (z) = \frac{1}{1 + e^{-z}}
−4 54.6 1/55.6=0.0181/55.6 = 0.018
−1 2.718 1/3.718=0.2691/3.718 = 0.269
0 1 1/2=1/2 = 0.5
1 0.368 1/1.368=0.7311/1.368 = 0.731
4 0.018 1/1.018=0.9821/1.018 = 0.982

What changed:

  • No sharp jump around the threshold −w0-w_{0}.
  • The output is a real number between 0 and 1, which we can read as a probability.
  • Instead of “like/dislike”, we get “probability of liking the movie”.

Everyday example A weather app says “70% chance of rain” instead of just “rain / no rain”. The sigmoid neuron answers in the same way.

4. Perceptron vs sigmoid neuron

Perceptron Sigmoid (logistic) neuron
Formula y=1y = 1 if ∑i=0nwixi≥0\sum _{i=0}^{n} w_{i}x_{i} \ge 0, else 0 y=11+exp⁡(−(w0+∑i=1nwixi))y = \frac{1}{1 + \exp (-(w_{0} + \sum _{i=1}^{n} w_{i}x_{i}))}
Output 0 or 1 any value between 0 and 1
Shape step smooth S-curve
Smooth? No Yes, continuous
Differentiable? No — not at the threshold −w0-w_{0} Yes, everywhere

Why “differentiable” matters: to learn weights with calculus, we need the slope (derivative) of the output with respect to each weight. A step has no useful slope (flat everywhere, a jump at one point). The sigmoid has a smooth slope everywhere. That is what gradient descent needs.

In one line: The sigmoid neuron is a perceptron with a smooth S-curve instead of a step; its output is a probability and it is differentiable.

Exam words: step function, sigmoid function, logistic function, σ(z)\sigma (z), smooth, continuous, differentiable, probability output, threshold −w0-w_{0}.

Quick check

  1. What is σ(0)\sigma (0)?
11+e0=11+1=0.5. \frac{1}{1 + e^{0}} = \frac{1}{1 + 1} = 0.5.
  1. A sigmoid neuron has w0=−0.5,w1=1w_{0} = -0.5, w_{1} = 1. Output for x1=0.5x_{1} = 0.5?
z=−0.5+1×0.5=0z = -0.5 + 1 \times 0.5 = 0. σ(0)=0.5\sigma (0) = 0.5. (Exactly at the old threshold the probability is 50%.)
  1. Why is the perceptron not differentiable?
Its output jumps suddenly from 0 to 1 at the threshold, so there is no slope at that point (and the slope is 0 everywhere else).
  1. Can the logistic output ever be exactly 0 or 1?
No. It gets very close to 0 for very negative z and very close to 1 for very positive z, but always stays strictly between them.
  1. σ(z)=0.731\sigma (z) = 0.731. What is σ(−z)\sigma (-z)?
1−0.731=0.2691 - 0.731 = 0.269. The sigmoid is symmetric: σ(−z)=1−σ(z)\sigma (-z) = 1 - \sigma (z).
YOUR STUDY PROGRESS

Finished this section? Save your progress.

Back to top ↑