SECTION 07 / 7

Examples: KL, JS and total variation

On this page 8 headings

KL divergence is generated by

f(u)=u\ln u

(with

f(0)=0

by continuity), giving

\begin{aligned}&D_{\mathrm{KL}}(P\,\|\,Q)\\&=\int p\ln\!\left(\frac{p}{q}\right)\end{aligned}

KL generally changes when P and Q are swapped; it is not a metric.

Illustration: Examples: KL, JS and total variation — Figure 21. Divergence versus metric distance.

Jensen–Shannon divergence is based on the mixture $M=\frac{P+Q}{2}$ :

\begin{aligned}&\operatorname{JS}(P,Q)\\&=\frac12\operatorname{KL}(P\|M)\\&+\frac12\operatorname{KL}(Q\|M)\end{aligned}

It is symmetric. A standard generating function for the corresponding f-divergence is

\begin{aligned}&f(u)\\&=\frac12\left[u\ln\frac{2u}{1+u}+\ln\frac{2}{1+u}\right]\end{aligned}

Total variation is

\operatorname{TV}(P,Q)=\frac12\int|p-q|

One generating function is

f(u)=\frac{|u-1|}{2}

Different conventions sometimes absorb the factor $\frac{1}{2}$ , so always check the definition used.

Worked example

Asymmetry of KL

Let $P=(0.9,0.1)$ , $Q=(0.5,0.5)$ .
$\begin{aligned}&\operatorname{KL}(P\|Q)\\&=0.9 \ln (1.8)\\&+0.1 \ln (0.2)\\&\approx 0.3681\end{aligned}$
$\begin{aligned}&\operatorname{KL}(Q\|P)\\&=0.5 \ln (\frac{0.5}{0.9})\\&+0.5 \ln (\frac{0.5}{0.1})\\&\approx 0.5108\end{aligned}$
The values differ.

Worked example

KL divergence in both directions

Take $P=(0.8,0.2)$ , $Q=(0.5,0.5)$ . Using natural logarithms,

\begin{aligned}&\operatorname{KL}(P\|Q)\\&=0.8 \ln (\frac{0.8}{0.5})\\&+0.2 \ln (\frac{0.2}{0.5})\\&=0.8 \ln (1.6)\\&+0.2 \ln (0.4)\\&\approx 0.1927\,\text{nats}\end{aligned}

Reverse the roles:

\begin{aligned}&\operatorname{KL}(Q\|P)\\&=0.5 \ln (\frac{0.5}{0.8})\\&+0.5 \ln (\frac{0.5}{0.2})\\&=0.5 \ln (0.625)\\&+0.5 \ln (2.5)\\&\approx 0.2231\,\text{nats}\end{aligned}

The log ratios and the probability weights both change, so the two directions need not agree.

Worked example

Jensen–Shannon divergence

Use the same $P=(0.8,0.2)$ and $Q=(0.5,0.5)$ . Step 1: form $M=\frac{P+Q}{2}=(0.65,0.35)$ . Step 2:

\begin{aligned}&\operatorname{KL}(P\|M)\\&=0.8 \ln (\frac{0.8}{0.65})\\&+0.2 \ln (\frac{0.2}{0.35})\\&\approx 0.0542\end{aligned}

Step 3:

\begin{aligned}&\operatorname{KL}(Q\|M)\\&=0.5 \ln (\frac{0.5}{0.65})\\&+0.5 \ln (\frac{0.5}{0.35})\\&\approx 0.0472\end{aligned}

Step 4:

\begin{aligned}&\operatorname{JS}(P,Q)\\&=\frac{0.0542+0.0472}{2}\\&\approx 0.0507\,\text{nats}\end{aligned}

Swapping P and Q leaves M unchanged and merely swaps the two KL terms, so JS is symmetric.

Worked example

Total variation

For $P=(0.8,0.2)$ and $Q=(0.5,0.5)$ ,

\begin{aligned}&\operatorname{TV}(P,Q)\\&=\frac{1}{2} (|0.8-0.5|+|0.2-0.5|)\\&=\frac{1}{2} (0.3+0.3)\\&=0.3\end{aligned}

The absolute values prevent positive and negative differences from canceling. Unlike KL and JS, the result here is not measured in nats.

Understand the idea

How to recognize which formula to use

KL: multiply each P probability by $\ln\frac{P}{Q}$ and add. JS: calculate the midpoint distribution M first, then average $\operatorname{KL}(P\|M)$ and $\operatorname{KL}(Q\|M)$ . TV: take absolute differences, sum and multiply by $\frac{1}{2}$ . All three can compare the same P and Q, but their numerical answers are not interchangeable.

Try it yourself

Practice with complete answers

Question 1: $P=(0.6,0.4)$ , $Q=(0.4,0.6)$ . Find TV.

Show solution

$\operatorname{TV}=\frac{1}{2}(|0.6-0.4|+|0.4-0.6|)=0.2$ .

Question 2: For the same P,Q, find $\operatorname{KL}(P\|Q)$ .

Show solution

$0.6 \ln (1.5)+0.4 \ln (\frac{2}{3})\approx 0.0811$ nats.

Question 3: Compute JS. $M=(0.5,0.5)$ ,

\begin{aligned}&\operatorname{KL}(P\|M)\\&=0.6 \ln (1.2)\\&+0.4 \ln (0.8)\\&\approx 0.02014\end{aligned}

$\operatorname{KL}(Q\|M)$ has the same value, so $\operatorname{JS}\approx 0.02014$ nats.

Understand the idea

Zero probabilities — careful calculation

For discrete KL, the term with $p_{i}=0$ contributes 0 by the standard limiting convention, provided $q_{i}>0$ . If $p_{i}>0$ but $q_{i}=0$ , $\operatorname{KL}(P\|Q)$ is infinite. This explains why blindly entering $\ln (0)$ or dividing by zero on a calculator gives misleading results. For JS, the mixture M avoids this particular infinite-KL issue when comparing finite discrete distributions.

Worked example

All three on the same distributions

Question. Let $P=(0.75,0.25)$ and $Q=(0.25,0.75)$ . Find $\operatorname{KL}(P\|Q)$ , $\operatorname{KL}(Q\|P)$ , $\operatorname{JS}(P,Q)$ , and $\operatorname{TV}(P,Q)$ .

\begin{aligned}&\operatorname{KL}(P\|Q)\\&=0.75 \ln (3)\\&+0.25 \ln (\frac{1}{3})\\&=(0.75-0.25)\ln 3\\&=0.5 \ln 3\\&\approx 0.5493\,\text{nats}\end{aligned}

\begin{aligned}&\operatorname{KL}(Q\|P)\\&=0.25 \ln (\frac{1}{3})\\&+0.75 \ln 3\\&=0.5 \ln 3\\&\approx 0.5493\,\text{nats}\end{aligned}

Here the two directions happen to be equal; KL is not symmetric in general.

$M=(0.5,0.5)$ .

\begin{aligned}&\operatorname{KL}(P\|M)\\&=0.75 \ln (1.5)\\&+0.25 \ln (0.5)\\&\approx 0.1308\end{aligned}

By symmetry of this particular example, $\operatorname{KL}(Q\|M)$ is the same. Hence $\operatorname{JS}(P,Q)\approx 0.1308$ nats.

\begin{aligned}&\operatorname{TV}(P,Q)\\&=\frac12\bigl(|0.75-0.25|\\&\qquad+|0.25-0.75|\bigr)\\&=\frac12(0.5+0.5)\\&=0.5\end{aligned}

Takeaway: the same P and Q can give different numerical divergences; always use the formula named in the question.

YOUR STUDY PROGRESS

Finished this section? Save your progress.