Keep pulling the thread on John Schulman.
An unbiased, low-variance estimator for `KL[q,p]` is given by the expression `(r - 1) - log(r)`, where `r = p(x)/q(x)`.
An unbiased, low-variance estimator for `KL[p,q]` is given by the expression `r * log(r) - (r - 1)`, where `r = p(x)/q(x)`.
The unbiased estimator `(r - 1) - log(r)` is a superior estimator for KL divergence compared to the biased estimator `1/2 * (log(r))^2`, as it has lower standard deviation in cases with larger KL values while remaining unbiased.
The Monte Carlo estimator `1/2 * (log(p(x)/q(x)))^2` for KL divergence has lower variance but is biased compared to the standard estimator `log(q(x)/p(x))`, which is unbiased but has high variance.
The low bias of the `1/2 * (log(p(x)/q(x)))^2` estimator for KL divergence is attributable to its expectation being an f-divergence.
All f-divergences with a differentiable function approximate KL divergence up to the second order when the two underlying probability distributions are close.
In an experiment where the true KL divergence between two normal distributions was 0.5, the biased estimator `1/2 * (log(r))^2` had a bias of 25% of the true value.