On Convergence Properties of Shannon Entropy

Abstract

Convergence properties of Shannon Entropy are studied. In the di erential setting, it is known that weak convergence of probability measures (convergence in distribution) is not enough for convergence of the associated di erential entropies. In that direction, an interesting example is introduced and discussed in light of new general results here provided for the desired di erential entropy convergence, results that take into account both compactly and uncompactly supported densities. Convergence of di erential entropy is also characterized in terms of the Kullback-Liebler discriminant for densities with fairly general supports, and it is shown that convergence in variation of probability measures guarantees such convergence under an appropriate boundedness condition on the densities involved. Results for the discrete setting are also provided, allowing for in nitely supported probability measures, by taking advantage of the equivalence between weak convergence and convergence in variation in that setting.