Lightweight binocular facial performance capture under uncontrolled lighting

  • Authors:
  • Levi Valgaerts;Chenglei Wu;Andrés Bruhn;Hans-Peter Seidel;Christian Theobalt

  • Affiliations:
  • MPI for Informatics;MPI for Informatics and Intel Visual Computing Institute;University of Stuttgart;MPI for Informatics;MPI for Informatics

  • Venue:
  • ACM Transactions on Graphics (TOG) - Proceedings of ACM SIGGRAPH Asia 2012
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

Recent progress in passive facial performance capture has shown impressively detailed results on highly articulated motion. However, most methods rely on complex multi-camera set-ups, controlled lighting or fiducial markers. This prevents them from being used in general environments, outdoor scenes, during live action on a film set, or by freelance animators and everyday users who want to capture their digital selves. In this paper, we therefore propose a lightweight passive facial performance capture approach that is able to reconstruct high-quality dynamic facial geometry from only a single pair of stereo cameras. Our method succeeds under uncontrolled and time-varying lighting, and also in outdoor scenes. Our approach builds upon and extends recent image-based scene flow computation, lighting estimation and shading-based refinement algorithms. It integrates them into a pipeline that is specifically tailored towards facial performance reconstruction from challenging binocular footage under uncontrolled lighting. In an experimental evaluation, the strong capabilities of our method become explicit: We achieve detailed and spatio-temporally coherent results for expressive facial motion in both indoor and outdoor scenes -- even from low quality input images recorded with a hand-held consumer stereo camera. We believe that our approach is the first to capture facial performances of such high quality from a single stereo rig and we demonstrate that it brings facial performance capture out of the studio, into the wild, and within the reach of everybody.