Reconstruction, registration, and modeling of deformable object shapes

• Authors: Jing Xiao;Takeo Kanade
• Venue: Reconstruction, registration, and modeling of deformable object shapes
• Year: 2005

Natural objects, such as expressive human faces, and dynamic scenes, such as cars running on the roads, generally vary the shapes as linear combinations of certain bases. With the expeditious development of computer and imaging technologies, the problems of reconstruction, registration, and modeling of such deformable shapes from image measurements has shown enormous importance for applications such as biomedical image interpretation and human computer interaction. Because the image measurements are generated by coupling two factors: non-rigid deformations and rigid similarity transformations between the shapes and the measuring systems, the essence of the three problems is to factorize the measurements and compute the deformable shapes (reconstruction), the rigid transformations (registration), and the shape bases (modeling). This thesis presents novel factorization algorithms for the three problems. First, we present a linear closed-form solution for reconstructing 3D deformable shapes from 2D images, assuming the weak-perspective camera model and non-degenerate cases. We prove that enforcing only the constraints on orthonormality of rigid rotations, as in the previous methods, inherently leads to ambiguous and invalid solutions. We show that the ambiguity stems from the non-uniqueness of the shape bases. We then introduce constraints on bases that resolve the ambiguity and result in a linear closed-form solution. We also develop methods for degeneracy cases and propose a 2-step factorization algorithm for uncalibrated perspective reconstruction. Second, we present a factorization-based technique for registering the deformable shapes in local measuring systems into a common coordinate system. This technique takes into account the shape deformation during the registration process and avoids the bias problem in the previous methods. Third, we apply the proposed algorithms to extracting the 3D shape model (consisting of the bases) of expressive human faces from monocular image sequences. Combining the 3D model with the 2D Active Appearance Model, we present a novel face model that describes the variations of both 2D and 3D face shapes and facial appearances. We then develop a real-time algorithm (60fps) that recovers the 2D and 3D face shapes, the 3D face poses, and the facial appearances by fitting the model to images.