The Minimum Description Length Principle (Adaptive Computation and Machine Learning)
The Minimum Description Length Principle (Adaptive Computation and Machine Learning)
Fisher information and stochastic complexity
IEEE Transactions on Information Theory
Information projections revisited
IEEE Transactions on Information Theory
| Hi-index | 0.00 |
The normalized maximum likelihood (NML) distribution plays a fundamental role in the MDL approach to statistical inference. It is only defined for statistical families with a finite Shtarkov sum. Here we characterize, for 1-dimensional exponential families, when the Shtarkov sum is finite. This turns out to be the case if and only if the minimax redundancy is finite, thus extending the reach of our results beyond the individual-sequence setting. In practice, the NML/Shtarkov distribution is often approximated by the Bayesian marginal distribution based on Jeffreys' prior. One serious problem is that in many cases Jeffreys' prior cannot be normalized. It has been conjectured that Jeffreys' prior cannot be normalized in exactly the cases where the Shtarkov sum is infinite, i.e, when the minimax redundancy and regret are infinite. We show that the conjecture is true for a large class of exponential families but that there exist examples where the conjecture is violated.