Mid-level features and spatio-temporal context for activity recognition

  • Authors:

  • Fei Yuan;Gui-Song Xia;Hichem Sahbi;Veronique Prinet

  • Affiliations:

  • Institute of Automation, Chinese Academy of Sciences, 1112, Zidonghua Building, 95 Zhongguancun East Road, Beijing 100190, PR China and LTCI CNRS, Telecom ParisTech, Paris, France;CNRS CEREMADE, University of Paris-Dauphine;LTCI CNRS, Telecom ParisTech, Paris, France;Institute of Automation, Chinese Academy of Sciences, 1112, Zidonghua Building, 95 Zhongguancun East Road, Beijing 100190, PR China

  • Venue:
  • Pattern Recognition
  • Year:
  • 2012

Quantified Score

Hi-index 0.01

Visualization

Abstract

Local spatio-temporal features have been shown to be effective and robust in order to represent simple actions. However, for high level human activities with long-range motion or multiple interactive body parts and persons, the limitation of low-level features blows up because of their localness. This paper addresses the problem by suggesting a framework that computes mid-level features and takes into account their contextual information. First, we represent human activities by a set of mid-level components, referred to as activity components, which have consistent structure and motion in spatial and temporal domain respectively. These activity components are extracted hierarchically from videos, i.e., extracting key-points, grouping them into trajectories and finally clustering trajectories into components. Second, to further exploit the interdependencies of the activity components, we introduce a spatio-temporal context kernel (STCK), which not only captures local properties of features but also considers their spatial and temporal context information. Experiments conducted on two challenging activity recognition datasets show that the proposed approach outperforms standard spatio-temporal features and our STCK context kernel improves further the performance.