On the entropy of a function (Q1028400)

Entropy is defined in such a way that, in appropriate limiting cases, the definition is consistent over different sampling regimes. For \(q>0\), the uniform quantization at level \(q\) of a measurable, real-valued and essentially bounded function \(f\) defined on \([0,1]\) partitions \([0,1]\) into the sets \(B_i=f^{-1}([iq, (i+1)q))\). One defines the entropy \(H_q(f)\) of \(f\) at quantization level \(q\) as \[ H_q (f) =-\sum_i \mu (B_i)\, \log_2\{\mu (B_i)\} \] where \(\mu\) denotes Lebesgue measure. To explain how this entropy depends jointly on sampling and quantization, first define the \(q\)-quantization operator of a sequence \(s\) by \[ Q_q(s)(j) = (i+1/2)q \qquad {\text{if}}\qquad iq\leq s(j)<(i+1)q \] and the (mid)pointwise and average sampling operators defined on functions by \[ S_n^p(f) (i)= f((2i+1)/2n) \qquad {\text{and}}\qquad S_n^a(f)(i) = \int_{i/n}^{(i+1)/n} f(t)\, dt. \] In the case of a continuous function, the main result can be stated as follows. Fix \(q>0\) and let \(Q_q S_n\) denote quantized sampling where \(S_n\) is \(S_n^p\) or \(S_n^a\). Let \(c_n (i)\) be the number of times that \(Q_q S_n (f)\) takes the value \((i+1/2)q\) and \(p_n(i)\) denote the relative probability of occurence of the value \(c_n(i)\), namely, \(p_n(i) = c_n(i)/i\). Then \(\lim_{n\to\infty} -\sum_i p_n(i) \log_2 p_n(i) = H_q(f)\). In particular, the sample entropy approaches the \(q\)-entropy regardless of whether pointwise sampling or average sampling is used. In the case of average sampling, \(f\) need not be continuous: it suffices that \(f\) is integrable.