User-assisted reflection detection and feature point tracking
Proceedings of the 10th European Conference on Visual Media Production
| Hi-index | 0.00 |
In this thesis we deal with separation of transparent layers in images and video. Our goal is to separate an input image or video sequence into their constituent scene layers. Transparent layer separation is an important problem which has attracted the attention of numerous researchers in recent years. The reason for that being that the world around us is full of transparent surfaces, such as windows, which are part of our natural scenery. Computer vision algorithms that strive to build a representation of natural scenes, such as, segmentation, classification, etc., are severly hampered by the superposition of transparent layers. Layer separation algorithms could serve at the pre-processing stage for such algorithms. In our research we have developed two algorithms for layer separation in images and video data. The first algorithm, the “Layer Information Exchange”, requires two input images of the same scene (each image having different proportions of each transparent layer). It achieves layer separation without prior assumptions on image formation model. Moreover, it handles spatially varying mixing of the underlying layers and varying illumination conditions. This algorithm can be used for separating transparent layers from single input video sequences. In this case one of the layers is assumed to have 2D parametric motion while the other can have any arbitrary nonrigid motion. The second algorithm enables the separation of transparent layers in video sequences where both layers are non-rigid, provided that one of them has an approximately repetitive behavior. Repetitive behaviors are very common for people, animals, and even natural phenomena, (e.g., running, walking, etc.). Moreover, repetitive behavior is not restricted to video sequences and can be found in other domains such as sound (e.g., repetitive tunes). For both algorithms we show results on synthetic and real data showing separation of transparent non-rigid layers for the first time. In addition we show the applicability of our approach for separating mixed audio signals from a single source. During our research we have explored numerous approaches, including using the information theoretic measure of Mutual Information. Our exploration led to a formalization of a generalized multivariate information measure. This generalized information measure provides a unified framework for many currently used and seemingly different information measures (including Mutual Information). In addition, our quest for optimization methods for functions based on probability distributions, led us to an approach which enables high-dimensional optimization search in the entropy space. This method might have merits for quantization, segmentation, and clustering problems in general.