Nonlinear Information Bottleneck

Author name not available (Why is that?)

Publication date: 5 May 2017

Abstract: Information bottleneck (IB) is a technique for extracting information in one random variable

X

that is relevant for predicting another random variable

Y

. IB works by encoding

X

in a compressed "bottleneck" random variable

M

from which

Y

can be accurately decoded. However, finding the optimal bottleneck variable involves a difficult optimization problem, which until recently has been considered for only two limited cases: discrete

X

and

Y

with small state spaces, and continuous

X

and

Y

with a Gaussian joint distribution (in which case optimal encoding and decoding maps are linear). We propose a method for performing IB on arbitrarily-distributed discrete and/or continuous

X

and

Y

, while allowing for nonlinear encoding and decoding maps. Our approach relies on a novel non-parametric upper bound for mutual information. We describe how to implement our method using neural networks. We then show that it achieves better performance than the recently-proposed "variational IB" method on several real-world datasets.

Has companion code repository: https://github.com/burklight/convex-IB-Lagrangian-PyTorch

This page was built for publication: Nonlinear Information Bottleneck

Report a bug (only for logged in users!)Click here to report a bug for this page (MaRDI item Q6286352)