An Introduction to Variational Methods for Graphical Models

  • Authors:

  • Michael I. Jordan;Zoubin Ghahramani;Tommi S. Jaakkola;Lawrence K. Saul

  • Affiliations:

  • Department of Electrical Engineering and Computer Sciences and Department of Statistics, University of California, Berkeley, CA 94720, USA. jordan@cs.berkeley.edu;Gatsby Computational Neuroscience Unit, University College London WC1N 3AR, UK. zoubin@gatsby.ucl.ac.uk;Artificial Intelligence Laboratory, MIT, Cambridge, MA 02139, USA. tommi@ai.mit.edu;AT&T Labs–Research, Florham Park, NJ 07932, USA. lsaul@research.att.edu

  • Venue:
  • Machine Learning
  • Year:
  • 1999

Quantified Score

Hi-index 0.11

Visualization

Abstract

This paper presents a tutorial introduction to the useof variational methods for inference and learning ingraphical models (Bayesian networks and Markov randomfields). We present a number of examples of graphical models, including the QMR-DT database, the sigmoid belief network, the Boltzmann machine, andseveral variants of hidden Markov models, in which it is infeasible to run exact inference algorithms. We then introduce variational methods, which exploit laws oflarge numbers to transform the original graphical model into a simplified graphical model in which inference isefficient. Inference in the simpified model provides bounds on probabilities of interest in the original model. We describe a general framework for generating variational transformations based on convex duality. Finally we return to the examples and demonstrate how variational algorithms can be formulated in each case.