Information Theory With Kernel Methods

DOI10.1109/TIT.2022.3211077arXiv2202.08545OpenAlexW4281396973MaRDI QIDQ6193556

Francis Bach

Publication date: 18 March 2024

Published in: IEEE Transactions on Information Theory (Search for Journal in Brave)

Abstract: We consider the analysis of probability distributions through their associated covariance operators from reproducing kernel Hilbert spaces. We show that the von Neumann entropy and relative entropy of these operators are intimately related to the usual notions of Shannon entropy and relative entropy, and share many of their properties. They come together with efficient estimation algorithms from various oracles on the probability distributions. We also consider product spaces and show that for tensor product kernels, we can define notions of mutual information and joint entropies, which can then characterize independence perfectly, but only partially conditional independence. We finally show how these new notions of relative entropy lead to new upper-bounds on log partition functions, that can be used together with convex optimization within variational inference methods, providing a new family of probabilistic inference methods.

Full work available at URL: https://arxiv.org/abs/2202.08545

zbMATH Keywords

quantum information theorykernel methods

Mathematics Subject Classification ID

Information and communication theory, circuits (94-XX)

Related Items (1)

Bridging algorithmic information theory and machine learning: a new approach to kernel learning

This page was built for publication: Information Theory With Kernel Methods

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6193556)