A unified definition of mutual information with applications in machine learning
type: Item
instance of: scholarly article
title: A unified definition of mutual information with applications in machine learning
zbMATH Open document ID: 1395.94204
DOI: 10.1155/2015/201874
author: Guoping Zeng
published in: Mathematical Problems in Engineering
publication date: 2018-08-27T00:00:00Z
review text: Summary: There are various definitions of mutual information. Essentially, these definitions can be divided into two classes: (1) definitions with random variables and (2) definitions with ensembles. However, there are some mathematical flaws in these definitions. For instance, Class 1 definitions either neglect the probability spaces or assume the two random variables have the same probability space. Class 2 definitions redefine marginal probabilities from the joint probabilities. In fact, the marginal probabilities are given from the ensembles and should not be redefined from the joint probabilities. Both Class 1 and Class 2 definitions assume a joint distribution exists. Yet, they all ignore an important fact that the joint or the joint probability measure is not unique. In this paper, we first present a new unified definition of mutual information to cover all the various definitions and to fix their mathematical flaws. Our idea is to define the joint distribution of two random variables by taking the marginal probabilities into consideration. Next, we establish some properties of the newly defined mutual information. We then propose a method to calculate mutual information in machine learning. Finally, we apply our newly defined mutual information to credit scoring.
Mathematics Subject Classification ID: 94A15
Mathematics Subject Classification ID: 62B10
Mathematics Subject Classification ID: 68T05
zbMATH DE Number: 6925750
Wikidata QID: Q59117586
MaRDI profile type: Publication
full work available at URL: https://doi.org/10.1155/2015/201874
OpenAlex ID: W2078028891
cites work: 10.1162/153244303322753616
cites work: A Mathematical Theory of Communication
cites work: Q5529067
cites work: Q3289420
cites work: Q5516510
cites work: Elements of Information Theory
cites work: Q5012903
cites work: Q5595624
cites work: Q4943605
cites work: Estimation of Entropy and Mutual Information
cites work: Metric divergence measures and information value in credit scoring
cites work: The Theory of Information and Coding
cites work: Greedy function approximation: A gradient boosting machine.
description: scientific article; zbMATH DE number 6925750