BSNotes.in — Notes for IIT Madras BS StudentsSearch your notes⌘ K
SECTION 06 / 8

The variational representation of f-divergence

2 min read
On this page 1 headings
Figure 43. Variational objective.
Figure 43. Variational objective.
Open image ↗
Illustration: The variational representation of f-divergence

Let P be the real distribution and Q the model distribution. Under the standard regularity and admissibility assumptions, the variational f-divergence representation is

Df(P∥Q)=sup⁡T{EP[T(X)]−EQ[f∗(T(X))]}\begin{aligned}&D_f(P\|Q)\\&=\sup_T\bigl\{\mathbb{E}_P[T(X)]\\&\qquad-\mathbb{E}_Q[f^*(T(X))]\bigr\}\end{aligned}

Here f is the generating convex function with

f(1)=0f(1)=0

and f∗f^* is its convex conjugate.

The first expectation is taken over real samples P. The second is taken over generated samples Q. Crucially, the SAME T appears in both terms, and f∗f^* acts on T(X) only in the second expectation.

Figure 42. Real and generated samples.
Figure 42. Real and generated samples.
Open image ↗
Illustration: The variational representation of f-divergence

How the formula arises: starting from

Df(P∥Q)=∫q(x)f ⁣(p(x)q(x)) dx\begin{aligned}&D_f(P\|Q)\\&=\int q(x)f\!\left(\frac{p(x)}{q(x)}\right)\,dx\end{aligned}

replace f(r) by sup⁡t{tr−f∗(t)}\sup_t\{tr-f^*(t)\}. For each x the ratio r=p(x)q(x)r=\frac{p(x)}{q(x)} can select a value T(x)T(x). Under appropriate conditions permitting the supremum to be exchanged with integration, this yields sup⁡T∫[p(x)T(x)−q(x)f∗(T(x))] dx\sup_T\int\left[p(x)T(x)-q(x)f^*(T(x))\right]\,dx, equal to the displayed difference of expectations.

A restriction to T\mathcal{T} gives sup⁡T∈TJ(T)≤Df(P∥Q)\sup_{T\in\mathcal{T}} J(T)\leq D_f(P\|Q), where

J(T)=EP[T]−EQ[f∗(T)]J(T)=\mathbb{E}_P[T]-\mathbb{E}_Q[f^*(T)]

provided T\mathcal{T} is contained in the admissible function class. Equality is possible if T\mathcal{T} is sufficiently expressive.

Figure 41. Restricted function class.
Figure 41. Restricted function class.
Open image ↗
Illustration: The variational representation of f-divergence
Worked example

Variational objective for KL

  1. For

    f(u)=uln⁡uf(u)=u\ln u

    the conjugate is

    f∗(t)=et−1f^*(t)=e^{t-1}

  2. Substitute into the general representation:

    KL⁡(P∥Q)=sup⁡T{EP[T(X)]−EQ[eT(X)−1]}\begin{aligned}&\operatorname{KL}(P\|Q)\\&=\sup_T\bigl\{\mathbb{E}_P[T(X)]\\&\qquad-\mathbb{E}_Q[e^{T(X)-1}]\bigr\}\end{aligned}

  3. The inner optimization searches for a suitable scalar-valued function T.

  4. With a restricted function family, maximizing the same expression yields a lower bound, not automatically the exact KL divergence.

YOUR STUDY PROGRESS

Finished this section? Save your progress.

Back to top ↑