Increasing and decreasing returns and losses in mutual information feature subset selection
type: Item
instance of: scholarly article
title: Increasing and decreasing returns and losses in mutual information feature subset selection
zbMATH Open document ID: 1229.62005
author: Gert Van Dijck
author: Marc M. Van Hulle
published in: Entropy
publication date: 2012-01-09T00:00:00Z
review text: Summary: Mutual information between a target variable and a feature subset is extensively used as a feature subset selection criterion. This work contributes to a more thorough understanding of the evolution of the mutual information as a function of the number of features selected. We describe decreasing returns and increasing returns behavior in sequential forward search and increasing losses and decreasing losses behavior in sequential backward search. We derive conditions under which the decreasing returns and the increasing losses behavior hold and prove the occurrence of this behavior in some Bayesian networks. The decreasing returns behavior implies that the mutual information is concave as a function of the number of features selected, whereas the increasing returns behavior implies this function is convex. The increasing returns and decreasing losses behavior are proven to occur in an XOR hypercube.
Mathematics Subject Classification ID: 62B10
Mathematics Subject Classification ID: 62F15
Mathematics Subject Classification ID: 68T35
zbMATH DE Number: 5995903
zbMATH Keywords: Bayesian networks
zbMATH Keywords: bit parity
zbMATH Keywords: conditional entropy
zbMATH Keywords: conditional mutual information
zbMATH Keywords: decreasing losses
zbMATH Keywords: feature subset selection
zbMATH Keywords: increasing losses
MaRDI profile type: Publication
full work available at URL: https://doi.org/10.3390/e12102144
OpenAlex ID: W1983091269
cites work: Computational Methods of Feature Selection
cites work: 10.1162/153244303322753616
cites work: Efficient selection of discriminative genes from microarray gene expression data for cancer diagnosis
cites work: Computer-Automated Design of Multifont Print Recognition Logic
cites work: A Programmed Algorithm for Designing Multifont Character Recognition Logics
cites work: Q3997653
cites work: Q3093235
cites work: Probability of error, equivocation, and the Chernoff bound
cites work: Relations between entropy and error probability
DOI: 10.3390/E12102144
description: scientific article; zbMATH DE number 5995903