High detail flexible viewpoint facial video from monocular input using static geometric proxies

  • Authors:

  • Markus Kettern;David Blumenthal-Barby;Peter Eisert

  • Affiliations:

  • Fraunhofer Heinrich Hertz Institute, Berlin, Germany;Fraunhofer Heinrich Hertz Institute, Berlin, Germany;Fraunhofer Heinrich Hertz Institute, Berlin, Germany

  • Venue:
  • Proceedings of the 6th International Conference on Computer Vision / Computer Graphics Collaboration Techniques and Applications
  • Year:
  • 2013

Quantified Score

Hi-index 0.00

Visualization

Abstract

We propose a method for creating flexible-viewpoint facial video from monocular input employing highly detailed static 3D reconstructions of an actor's head. We use the term flexible-viewpoint to indicate that the viewpoint can be arbitrarily chosen (as in free-viewpoint video), but from a restricted set of viewing directions. Our method enables dynamic changes of the viewpoint without requiring estimation of the head geometry form the video sequences which is hard with the methods typically used for creating free-viewpoint video. Alongside video capture of a certain actor, we record static high resolution stereo images of the actor's head and face. From these images, we create a detailed 3D model of the head by image-based reconstruction methods. We propose two methods to register this 3D model to the a starting frame of the video stream following the actors head and facial action. Furthermore, we show how model-based tracking over the whole video sequence provides precise head pose estimates for each video frame. Once the registration is complete, the 3D model serves as geometric proxy for image-based rendering techniques in order to create novel viewpoints using the video stream as texture.