Automatic Face Modeling from Monocular Image Sequences Using Modified Non Parametric Regression and an Affine Camera Model

• Authors: K. Sengupta;W. Shiqin;C. C. Ko;P. Burman
• Venue: FG '00 Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000
• Year: 2000

In this paper, we present the theory of modified non parametric regression for estimating the 3D face structure of a human from a monocular image sequence. In the preprocessing stage, the face region is segmented from the background using both color and motion information, by using a hierarchical block motion estimation method. By using the affine camera projection geometry, and a given choice of an image frame pair in the sequence, we adopt the KvD model to express the depth at each point on the face region as a function of the unknown out of plane rotation, and some measurable quantities computed directly from the optical flow. This is repeated for multiple image pairs (keeping one fixed image frame which we formally call the "base" image, and choosing another frame from the sequence). The true depth map is next estimated from these equations using a modified non parametric regression technique, and this forms the core contribution of this paper. We conducted experiments on various image sequences to verify the effectiveness of the technique, and propose to extend it for photo-realistic modeling of arbitrary (non-face) objects from image sequences.