Approximation of functions over redundant dictionaries using coherence

• Authors: Anna C. Gilbert;S. Muthukrishnan;Martin J. Strauss
• Venue: SODA '03 Proceedings of the fourteenth annual ACM-SIAM symposium on Discrete algorithms
• Year: 2003

One of the central problems of modern mathematical approximation theory is to approximate functions, or signals, concisely, with elements from a large candidate set called a dictionary. Formally, we are given a signal A ∈ RN and a dictionary D = {φi}i∈I of unit vectors that span RN. A representation R of B terms for input A ∈ RN is a linear combination of dictionary elements, R = σi∈A αiφi, for φi ∈ D and some A, |A| ≥ B. Typically, B ⪡ N, so that R is a concise approximation to signal A. The error of the representation indicates by how well it approximates A, and is given by ∥A - R∥2 = √Σt|A[t - R[t]|2. The problem is to find the best B-term representation, i.e., find a R that minimizes ∥A - R∥2. A dictionary may be redundant in the sense that there is more than one possible exact representation for A, i.e., |D| N = dim(RN). Redundant dictionaries are used because, both theoretically and in practice, for important classes of signals, as the size of a dictionary increases, the error and the conciseness of the approximations improve.We present the first known efficient algorithm for finding a provably approximate representation for an input signal over redundant dictionaries. We identify and focus on redundant dictionaries with small coherence (ie., vectors are nearly orthogonal). We present an algorithm that preprocesses any such dictionary in time and space polynomial in |D|, and obtains an 1 + ε approximate representation of the given signal in time nearly linear in signal size N and polylogarithmic in |D|; by contrast, most algorithms in the literature require Ω(|D|)time, and, yet, provide no provable bounds. The technical crux of our result is our proof that two commonly used local search techniques, when combined appropriately, gives a provably near-optimal signal representation over redundant dictionaries with small coherence. Our result immediately applies to several specific redundant dictionaries considered by the domain experts thus far. In addition, we present new redundant dictionaries which have small coherence (and therefore are amenable to our algorithms) and yet have significantly large sizes, thereby adding to the redundant dictionary construction literature.Work with redundant dictionaries forms the emerging field of highly nonlinear approximation theory. We have presented algorithmic results for some of the most basic problems in this area, but other mathematical and algorithmic questions remain to be explored.