The minimax distortion redundancy in noisy source coding

• Authors: A. Dembo;T. Weissman
• Venue: IEEE Transactions on Information Theory
• Year: 2006

Consider the problem of finite-rate filtering of a discrete memoryless process {Xi}i≥1 based on its noisy observation sequence {Zi}i≥1, which is the output of a discrete memoryless channel (DMC) whose input is {Xi}i≥1. When the distribution of the pairs (Xi,Zi), PX,Z, is known, and for a given distortion measure, the solution to this problem is well known to be given by classical rate-distortion theory upon the introduction of a modified distortion measure. We address the case where PX,Z, rather than being completely specified, is only known to belong to some set Λ. For a fixed encoding rate R, we look at the worst case, over all θ∈Λ, of the difference between the expected distortion of a given scheme which is not allowed to depend on the active source θ∈Λ and the value of the distortion-rate function at R corresponding to the noisy source θ. We study the minimum attainable value achievable by any scheme operating at rate R for this worst case quantity, denoted by D(Λ, R). Linking this problem and that of source coding under several distortion measures, we prove a coding theorem for the latter problem and apply it to characterize D(Λ, R) for the case where all members of Λ share the same noisy marginal. For the case of a general Λ, we obtain a single-letter characterization of D(Λ, R) for the finite-alphabet case. This gives, in particular, a necessary and sufficient condition on the set Λ for the existence of a coding scheme which is universally optimal for all members of Λ and characterizes the approximation-estimation tradeoff for statistical modeling of noisy source coding problems. Finally, we obtain D(Λ, R) in closed form for cases where Λ consists of distributions on the (channel) input-output pair of a Bernoulli source corrupted by a binary-symmetric channel (BSC). In particular, for the case where Λ consists of two sources: the all-zero source corrupted by a BSC with crossover probability r and the Bernoulli(r) source with a noise-free channel; we find that universality be- comes increasingly hard with increasing rate.