Predicting a binary sequence almost as well as the optimal biased coin
COLT '96 Proceedings of the ninth annual conference on Computational learning theory
Generalized Shannon Code Minimizes the Maximal Redundancy
LATIN '02 Proceedings of the 5th Latin American Symposium on Theoretical Informatics
How to Achieve Minimax Expected Kullback-Leibler Distance from an Unknown Finite Distribution
ALT '02 Proceedings of the 13th International Conference on Algorithmic Learning Theory
Predicting a binary sequence almost as well as the optimal biased coin
Information and Computation
Coding on countably infinite alphabets
IEEE Transactions on Information Theory
Relative loss bounds for on-line density estimation with the exponential family of distributions
UAI'99 Proceedings of the Fifteenth conference on Uncertainty in artificial intelligence
| Hi-index | 754.90 |
Let Xn=(X1,...,Xn) be a memoryless source with unknown distribution on a finite alphabet of size k. We identify the asymptotic minimax coding redundancy for this class of sources, and provide a sequence of asymptotically minimax codes. Equivalently, we determine the limiting behavior of the minimax relative entropy minQXn maxpXn D(PXn||QXn), where the maximum is over all independent and identically distributed (i.i.d.) source distributions and the minimum is over all joint distributions. We show in this paper that the minimax redundancy minus ((k-1)/2) log(n/(2πe)) converges to log∫√(det I(θ))dθ=log (Γ(1/2)k/Γ(k/2)), where I(θ) is the Fisher information and the integral is over the whole probability simplex. The Bayes strategy using Jeffreys' prior is shown to be asymptotically maximin but not asymptotically minimax in our setting. The boundary risk using Jeffreys' prior is higher than that of interior points. We provide a sequence of modifications of Jeffreys' prior that put some prior mass near the boundaries of the probability simplex to pull down that risk to the asymptotic minimax level in the limit