Robust and Fast Measure of Information via Low-rank Representation

arXiv2211.16784MaRDI QIDQ6419010

Author name not available (Why is that?)

Publication date: 30 November 2022

Abstract: The matrix-based R'enyi's entropy allows us to directly quantify information measures from given data, without explicit estimation of the underlying probability distribution. This intriguing property makes it widely applied in statistical inference and machine learning tasks. However, this information theoretical quantity is not robust against noise in the data, and is computationally prohibitive in large-scale applications. To address these issues, we propose a novel measure of information, termed low-rank matrix-based R'enyi's entropy, based on low-rank representations of infinitely divisible kernel matrices. The proposed entropy functional inherits the specialty of of the original definition to directly quantify information from data, but enjoys additional advantages including robustness and effective calculation. Specifically, our low-rank variant is more sensitive to informative perturbations induced by changes in underlying distributions, while being insensitive to uninformative ones caused by noises. Moreover, low-rank R'enyi's entropy can be efficiently approximated by random projection and Lanczos iteration techniques, reducing the overall complexity from $mathcalO(n^3)$ to $mathcalO(n^{2}s)$ or even $mathcalO(n{s}^2)$, where $n$ is the number of data samples and $slln$. We conduct large-scale experiments to evaluate the effectiveness of this new information measure, demonstrating superior results compared to matrix-based R'enyi's entropy in terms of both performance and computational efficiency.

Has companion code repository: https://github.com/gamepiaynmo/lrmi

This page was built for publication: Robust and Fast Measure of Information via Low-rank Representation

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6419010)