Reconstructing detailed dynamic face geometry from monocular video

  • Authors:
  • Pablo Garrido;Levi Valgaert;Chenglei Wu;Christian Theobalt

  • Affiliations:
  • Max Planck Institute for Informatics;Max Planck Institute for Informatics;Max Planck Institute for Informatics and Intel Visual Computing Institute;Max Planck Institute for Informatics

  • Venue:
  • ACM Transactions on Graphics (TOG)
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

Detailed facial performance geometry can be reconstructed using dense camera and light setups in controlled studios. However, a wide range of important applications cannot employ these approaches, including all movie productions shot from a single principal camera. For post-production, these require dynamic monocular face capture for appearance modification. We present a new method for capturing face geometry from monocular video. Our approach captures detailed, dynamic, spatio-temporally coherent 3D face geometry without the need for markers. It works under uncontrolled lighting, and it successfully reconstructs expressive motion including high-frequency face detail such as folds and laugh lines. After simple manual initialization, the capturing process is fully automatic, which makes it versatile, lightweight and easy-to-deploy. Our approach tracks accurate sparse 2D features between automatically selected key frames to animate a parametric blend shape model, which is further refined in pose, expression and shape by temporally coherent optical flow and photometric stereo. We demonstrate performance capture results for long and complex face sequences captured indoors and outdoors, and we exemplify the relevance of our approach as an enabling technology for model-based face editing in movies and video, such as adding new facial textures, as well as a step towards enabling everyone to do facial performance capture with a single affordable camera.