A Bayesian characterization of relative entropy
type: Item
instance of: scholarly article
zbMATH Open document ID: 1321.94023
author: John Baez
author: Tobias Fritz
publication date: 2014-08-25T00:00:00Z
full work available at URL: https://arxiv.org/abs/1402.3067
full work available at URL: 29-16abs.html
Mathematics Subject Classification ID: 94A17
Mathematics Subject Classification ID: 62F15
Mathematics Subject Classification ID: 18B99
zbMATH DE Number: 6334034
zbMATH Keywords: relative entropy
zbMATH Keywords: Kullback-Leibler divergence
zbMATH Keywords: measures of information
MaRDI profile type: Publication
arXiv ID: 1402.3067
arXiv classification: cs.IT
arXiv classification: math-ph
arXiv classification: math.IT
arXiv classification: math.MP
arXiv classification: math.PR
arXiv classification: quant-ph
title: A Bayesian characterization of relative entropy
published in: Theory and Applications of Categories
review text: This paper gives a new characterization of the concept of relative entropy, aka \textit{relative information}, \textit{relative gain} or \textit{Kullback-Leibler divergence}. Whenever we have two probability distributions \(p\) and \(q\) on the same set \(X\), we define the information of \(q\) relative to \(p\) asNEWLINE\[CARRIAGE_RETURNNEWLINES(q,p)=\sum_{x\in X}q_{x}\ln\left( \frac{q_{x}}{p_{x}}\right)CARRIAGE_RETURNNEWLINE\]NEWLINEwhere \(q_{x}\ln\left( \frac{q_{x}}{p_{x}}\right) \) is set equal to \(\infty\) when \(p_{x}=0\), unless \(q_{x}\) is also \(0\), in which case it is set equal to \(0\).NEWLINENEWLINEBayesian probability theory emphasizes the role of the prior so that relative entropy naturally lends itself to a Bayesian interpretation [\textit{P. Baldi} and \textit{L. Itti}, Neural Netw. 23, No. 5, 649--666 (2010; Zbl 1401.62225)]. The goal of this paper is to make this precise in a mathematical characterization of relative entropy. The authors consider a category \(\mathtt{FinStat}\), where an object \((X,q)\) is a finite set \(X\) gifted with a probability distribution \(x\mapsto q_{x}\), while a morphism \((f,s):(X,q)\rightarrow(Y,r)\) is a measure-preserving function \(f:X\rightarrow Y\) hand in hand with a probability distribution \(x\mapsto s_{xy}\) on \(X\) for each element \(y\in Y\) with the property \(s_{xy}=0\) unless \(f(x)=y\). NEWLINENEWLINEIntuitively speaking, an object of \(\mathtt{FinStat}\) is to be thought of a system with some finite set of states as well as a probanility distribution on it. A morphism \((f,s):(X,q)\rightarrow(Y,r)\) is a deterministic measuring process \(f:X\rightarrow Y\) mapping states of some system under measurement to those of a measuring apparatus as well as a hypothesis \(s\) meaning the probability \(s_{xy}\) that the system under measurement is in the state \(x\) given any measurement outcome \(y\in Y\).NEWLINENEWLINEGiven a morphism \((f,s):(X,q)\rightarrow(Y,r)\) in \(\mathtt{FinStat}\), the authors de…
reviewed by: Hirokazu Nishimura
description: scientific article; zbMATH DE number 6334034