BSNotes.in — Notes for IIT Madras BS StudentsSearch your notes⌘ K
SECTION 06 / 7

Why convexity enters f-divergence

3 min read
On this page 6 headings

The example introduces variational divergence minimization through f-divergences. Under the standard density-ratio formulation, let P and Q have densities p and q, and assume P is absolutely continuous with respect to Q. For a convex generating function f on nonnegative ratios, normalized by

f(1)=0f(1)=0
Figure 20. Convexity and divergence nonnegativity.
Figure 20. Convexity and divergence nonnegativity.
Open image ↗
Illustration: Why convexity enters f-divergence
Df(P ∥ Q)=∫q(x) f ⁣(p(x)q(x)) dx\begin{aligned}&D_f(P\,\|\,Q)\\&=\int q(x)\,f\!\left(\frac{p(x)}{q(x)}\right)\,dx\end{aligned}

For the calculations below, we first use discrete distributions with positive Q probabilities. At zero-probability outcomes, apply the standard limiting conventions rather than dividing by zero.

Understand the idea

Why is DfD_{f} nonnegative?

Under QQ, define the density ratio:

R=p(X)q(X)R=\frac{p(X)}{q(X)}

Its expectation under QQ is one:

EQ[R]=∫q(x) p(x)q(x) dx=∫p(x) dx=1\begin{aligned}&\mathbb{E}_Q[R]\\&=\int q(x)\,\frac{p(x)}{q(x)}\,dx\\&=\int p(x)\,dx\\&=1\end{aligned}

Apply Jensen’s inequality to the convex function ff:

EQ[f(R)]≥f ⁣(EQ[R])=f(1)=0.\begin{aligned}\mathbb{E}_Q[f(R)]&\geq f\!\left(\mathbb{E}_Q[R]\right)\\&=f(1)\\&=0.\end{aligned}

Therefore:

Df(P ∥ Q)=EQ ⁣[f ⁣(p(X)q(X))]≥0\begin{aligned}&D_f(P\,\|\,Q)\\&=\mathbb{E}_Q\!\left[f\!\left(\frac{p(X)}{q(X)}\right)\right]\geq0\end{aligned}

Equality is guaranteed when P=QP=Q; the converse needs stronger assumptions on ff.

Worked example

Discrete f-divergence example

  1. Let P=(34,14)P = (\frac{3}{4}, \frac{1}{4}), Q=(12,12)Q = (\frac{1}{2}, \frac{1}{2}), and

    f(u)=uln⁡uf(u)=u\ln u

  2. Ratios are (32,12)(\frac{3}{2}, \frac{1}{2}). Then

    Df(P∥Q)=(34)ln⁡(32)+(14)ln⁡(12)\begin{aligned}&D_{f}(P\|Q)\\&= (\frac{3}{4})\ln (\frac{3}{2})\\&+ (\frac{1}{4})\ln (\frac{1}{2})\end{aligned}

  3. Numerically this is about 0.1308 nats, which is nonnegative.

Understand the idea

Reading the f-divergence formula without skipping steps

For a discrete sample space, replace the integral by a sum:

Df(P ∥ Q)=∑iqif ⁣(piqi)\begin{aligned}&D_f(P\,\|\,Q)\\&=\sum_i q_i f\!\left(\frac{p_i}{q_i}\right)\end{aligned}

First choose the outcomes, then write the pip_{i} and qiq_{i} values, calculate the ratios piqi\frac{p_i}{q_i}, apply f to each ratio, multiply by qiq_{i}, and add. This is the same structure as the integral in the chapter, expressed as a small table of probabilities.

Worked example

A different convex f

Choose

f(u)=(u−1)2f(u)=(u-1)^2

P=(34,14)P=(\frac{3}{4},\frac{1}{4}) and Q=(12,12)Q=(\frac{1}{2},\frac{1}{2}). The ratios are 1.5 and 0.5. Their f-values are

(0.5)2=0.25(0.5)^{2}=0.25

and

(−0.5)2=0.25(-0.5)^{2}=0.25

Multiply each by its Q weight: (12)(0.25)+(12)(0.25)=0.25(\frac{1}{2})(0.25)+(\frac{1}{2})(0.25)=0.25. This is an illustration of the general f-divergence calculation, not an additional named divergence for the syllabus.

Watch out

Conditions and common errors

The usual ratio formula requires care where q(x)=0q(x)=0; do not divide by zero. For introductory finite examples we use strictly positive Q probabilities. Also, the function f must satisfy

f(1)=0f(1)=0

and the convexity condition. If P=QP=Q, every ratio equals 1, so

Df(P ∥ P)=∑ipif(1)=0D_f(P\,\|\,P)=\sum_i p_i f(1)=0
Understand the idea

A complete calculation recipe for finite distributions

Step 1 — Check that P and Q are probability distributions: each entry is nonnegative and each row sums to 1. Both must describe the same outcomes in the same order.

Step 2 — For KL, compute the ratios piqi\frac{p_i}{q_i} and then the contributions piln⁡piqip_i\ln\frac{p_i}{q_i}. Sum the contributions. Use natural logarithms unless the question specifies another base.

Step 3 — For JS, form mi=pi+qi2m_i=\frac{p_i+q_i}{2} for each outcome. Compute KL⁡(P∥M)\operatorname{KL}(P\|M) and KL⁡(Q∥M)\operatorname{KL}(Q\|M) separately, then take their average.

Step 4 — For total variation, compute ∣pi−qi∣|p_i-q_i| for each outcome, sum, and divide by two. There are no logarithms.

Final check — KL and JS are nonnegative; total variation lies between 0 and 1. A negative numerical KL usually signals an arithmetic error.

YOUR STUDY PROGRESS

Finished this section? Save your progress.

Back to top ↑