L(1)-norm sparse bayesian learning: theory and applications

• Authors: Daniel D. Lee;Yuanqing Lin
• Venue: L(1)-norm sparse bayesian learning: theory and applications
• Year: 2008

The elements in the real world are often sparsely connected. For example, in a social network, each individual is only sparsely connected to a small portion of people in the network; for certain disease (like breast cancer), even though human have tens of thousands of genes, only a small number of them are connected to the disease; for a filter modeling an acoustic room impulse response, only a small portion of filter coefficients are nonzero. Discovering the sparse representations of the real world is important since they provide not only the neatest insight for understanding the world but also the most efficient way for changing the world. Therefore, finding sparse representations has attracted a great amount of research effort in the past decade, and it has been a driving force for many exciting new fields, such as sparse coding and compressive sampling. The research effort on finding sparse representations has covered both theories and applications. In its theoretic aspects, researchers have developed many approaches (such as nonnegative constraint, l1 -norm sparsity regularization and sparse Bayesian learning with independent Gaussian prior) for encouraging sparse solutions and established some conditions under which the true solutions (which are sparse) could be found by those approaches. Meanwhile, finding sparse representations has found its applications in a wide spectrum of fields such as acoustic/image signal processing, computer vision, natural language processing, bioinformatics, finance modeling, and so on. However, despite of the intense studies in finding sparse solutions in the last decade, there is a fundamental issue still remained almost untouched, that is, how sparse is the optimally sparse in representing given data? This thesis aims to answer the above fundamental question by establishing a theory of l1-norm sparse Bayesian learning . In particular, using l1-norm regularized least squares as an example, we show how the l1-norm sparse Bayesian learning extends the conventional uniform l 1-norm sparsity regularization, where all variables desired to be sparse share a single scalar regularization parameter, to independent l1-norm sparsity regularization, where each variable is associated with an independent regularization parameter. In the independent l1-norm sparsity regularization, the optimal sparseness of solutions is then fully defined in a Bayesian sense via the optimal l1-norm sparsity regularization parameters and inferred by learning directly from data. This is why we call our Bayesian approach sparse learning, which is very different from conventional methods where there is only single l1-norm regularization parameter and it is determined by ad-hoc manners (like cross-validation). The proposed l1-norm sparse Bayesian learning shows superior performance in both simulations and real examples. Our simulation results demonstrate that the l1-norm sparse Bayesian learning is able to accurately resolve the true sparseness in solutions even in very noisy data, and it provides better performance than the conventional uniform l1-norm regularization and l 2-norm Bayesian sparse learning (also known as relevance vector machine). In real examples, we show the l1-norm sparse Bayesian learning is effective for speech dereverberation and acoustic time different of arrival (TDOA) estimation in reverberant environments, both of which are hard problems and have remained open problems after a long history of research.