Automatic Analysis of Multimodal Group Actions in Meetings

Authors:
Iain McCowan;Daniel Gatica-Perez;Samy Bengio;Guillaume Lathoud;Mark Barnard;Dong Zhang
Affiliations:
IEEE;IEEE;IEEE;IEEE;IEEE;-
Venue:
IEEE Transactions on Pattern Analysis and Machine Intelligence
Year:
2005

Citing 13
Cited 78

Fundamentals of speech recognition

Fundamentals of speech recognition
Rough'n'Ready: a meeting recorder and browser

ACM Computing Surveys (CSUR)
A Bayesian Computer Vision System for Modeling Human Interactions

IEEE Transactions on Pattern Analysis and Machine Intelligence
Multi-stream adaptive evidence combination for noise robust ASR

Speech Communication - Special issue on noise robust ASR
Statistical color models with application to skin detection

International Journal of Computer Vision
Distributed meetings: a meeting capture and broadcasting system

Proceedings of the tenth ACM international conference on Multimedia
The Acquisition and Use of Interaction Behavior Models

CVPR '98 Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition
Modeling Individual and Group Actions in Meetings: A Two-Layer HMM Framework

CVPRW '04 Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 7 - Volume 07
The meeting project at ICSI

HLT '01 Proceedings of the first international conference on Human language technology research
Detection of agreement vs. disagreement in meetings: training with unlabeled data

NAACL-Short '03 Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology: companion volume of the Proceedings of HLT-NAACL 2003--short papers - Volume 2
The KidsRoom: A Perceptually-Based Interactive and Immersive Story Environment

Presence: Teleoperators and Virtual Environments
Content-based video indexing of TV broadcast news using hidden Markov models

ICASSP '99 Proceedings of the Acoustics, Speech, and Signal Processing, 1999. on 1999 IEEE International Conference - Volume 06
Audio-visual speech modeling for continuous speech recognition

IEEE Transactions on Multimedia

Towards automatic analysis of social interaction patterns in a nursing home environment from video

Proceedings of the 6th ACM SIGMM international workshop on Multimedia information retrieval
Multimodal detection of human interaction events in a nursing home environment

Proceedings of the 6th international conference on Multimodal interfaces
Automatic detection of interaction groups

ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
A probabilistic inference of multiparty-conversation structure based on Markov-switching models of gaze patterns, head directions, and utterances

ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
Extracting information from multimedia meeting collections

Proceedings of the 7th ACM SIGMM international workshop on Multimedia information retrieval
Détection automatique des groupes d'interactions

UbiMob '05 Proceedings of the 2nd French-speaking conference on Mobility and ubiquity computing
Quantifying interpersonal influence in face-to-face conversations based on visual attention patterns

CHI '06 Extended Abstracts on Human Factors in Computing Systems
The influence of cross-validation on video classification performance

MULTIMEDIA '06 Proceedings of the 14th annual ACM international conference on Multimedia
Human-centered computing: a multimedia perspective

MULTIMEDIA '06 Proceedings of the 14th annual ACM international conference on Multimedia
Automatic detection of group functional roles in face to face interactions

Proceedings of the 8th international conference on Multimodal interfaces
Prototyping novel collaborative multimodal systems: simulation, data collection and analysis tools for the next decade

Proceedings of the 8th international conference on Multimodal interfaces
Detecting social interactions of the elderly in a nursing home environment

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP)
Multimodal human-computer interaction: A survey

Computer Vision and Image Understanding
Multimedia and human-in-the-loop: interaction as content enrichment

Proceedings of the international workshop on Human-centered multimedia
Audio-visual multi-person tracking and identification for smart environments

Proceedings of the 15th international conference on Multimedia
Towards smart meeting: enabling technologies and a real-world application

Proceedings of the 9th international conference on Multimodal interfaces
Automatic inference of cross-modal nonverbal interactions in multiparty conversations: "who responds to whom, when, and how?" from gaze, head gestures, and utterances

Proceedings of the 9th international conference on Multimodal interfaces
Using the influence model to recognize functional roles in meetings

Proceedings of the 9th international conference on Multimodal interfaces
A writer identification system for on-line whiteboard data

Pattern Recognition
HCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces

HCI Beyond the GUI: Design for Haptic, Speech, Olfactory, and Other Nontraditional Interfaces
Context-aware computing for assistive meeting system

Proceedings of the 1st international conference on PErvasive Technologies Related to Assistive Environments
Audiovisual integration with Segment Models for tennis video parsing

Computer Vision and Image Understanding
Daily Routine Classification from Mobile Phone Data

MLMI '08 Proceedings of the 5th international workshop on Machine Learning for Multimodal Interaction
Role recognition in multiparty recordings using social affiliation networks and discrete distributions

ICMI '08 Proceedings of the 10th international conference on Multimodal interfaces
Social signals, their function, and automatic analysis: a survey

ICMI '08 Proceedings of the 10th international conference on Multimodal interfaces
Probabilistic integration of sparse audio-visual cues for identity tracking

MM '08 Proceedings of the 16th ACM international conference on Multimedia
Social signal processing: state-of-the-art and future perspectives of an emerging domain

MM '08 Proceedings of the 16th ACM international conference on Multimedia
Hierarchical group process representation in multi-agent activity recognition

Image Communication
Multimodal identity tracking in a smart room

Personal and Ubiquitous Computing
Detecting small group activities from multimodal observations

Applied Intelligence
Review: Ambient intelligence: Technologies, applications, and opportunities

Pervasive and Mobile Computing
Social signal processing: Survey of an emerging domain

Image and Vision Computing
Automatic nonverbal analysis of social interaction in small groups: A review

Image and Vision Computing
Review: The use of pervasive sensing for behaviour profiling - a survey

Pervasive and Mobile Computing
A speaker diarization method based on the probabilistic fusion of audio-visual location information

Proceedings of the 2009 international conference on Multimodal interfaces
Classification of patient case discussions through analysis of vocalisation graphs

Proceedings of the 2009 international conference on Multimodal interfaces
Modeling dominance in group conversations using nonverbal activity cues

IEEE Transactions on Audio, Speech, and Language Processing - Special issue on multimodal processing in speech-based interactions
Off-line handwritten word recognition using multi-stream hidden Markov models

Pattern Recognition Letters
Group interaction analysis in dynamic context

IEEE Transactions on Systems, Man, and Cybernetics, Part B: Cybernetics - Special issue on human computing
Smart meeting systems: A survey of state-of-the-art and open issues

ACM Computing Surveys (CSUR)
Analysis of multi-agent activity using petri nets

Pattern Recognition
Preference model assisted activity recognition learning in a smart home environment

IROS'09 Proceedings of the 2009 IEEE/RSJ international conference on Intelligent robots and systems
A survey of context modelling and reasoning techniques

Pervasive and Mobile Computing
An adaptive vision system toward implicit human computer interaction

UAHCI'07 Proceedings of the 4th international conference on Universal access in human-computer interaction: ambient interaction
Online analysis of hierarchical events in meetings

UI-HCII'07 Proceedings of the 2nd international conference on Usability and internationalization
Robust real time face tracking for the analysis of human behaviour

MLMI'07 Proceedings of the 4th international conference on Machine learning for multimodal interaction
Meeting state recognition from visual and aural labels

MLMI'07 Proceedings of the 4th international conference on Machine learning for multimodal interaction
Automatic decision detection in meeting speech

MLMI'07 Proceedings of the 4th international conference on Machine learning for multimodal interaction
Recognition and understanding of meetings

HLT '10 Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics
An interaction-embedded HMM framework for human behavior understanding: with nursing environments as examples

IEEE Transactions on Information Technology in Biomedicine
What is happening now? Detection of activities of daily living from simple visual features

Personal and Ubiquitous Computing
Automatic detection of dominance and expected interest

EURASIP Journal on Advances in Signal Processing
Strategies for inference mechanism of conditional random fields for multiple-resident activity recognition in a smart home

IEA/AIE'10 Proceedings of the 23rd international conference on Industrial engineering and other applications of applied intelligent systems - Volume Part I
Generative modeling and classification of dialogs by a low-level turn-taking feature

Pattern Recognition
Discovering multirelational structure in social media streams

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP)
Functional brain imaging with multi-objective multi-modal evolutionary optimization

PPSN'06 Proceedings of the 9th international conference on Parallel Problem Solving from Nature
VACE multimodal meeting corpus

MLMI'05 Proceedings of the Second international conference on Machine Learning for Multimodal Interaction
Multimodal integration for meeting group action segmentation and recognition

MLMI'05 Proceedings of the Second international conference on Machine Learning for Multimodal Interaction
A multimodal discourse ontology for meeting understanding

MLMI'05 Proceedings of the Second international conference on Machine Learning for Multimodal Interaction
Review: Situation identification techniques in pervasive computing: A review

Pervasive and Mobile Computing
Inferring competitive role patterns in reality TV show through nonverbal analysis

Multimedia Tools and Applications
Extracting activities from multimodal observation

KES'06 Proceedings of the 10th international conference on Knowledge-Based Intelligent Information and Engineering Systems - Volume Part II
Multimodal human computer interaction: a survey

ICCV'05 Proceedings of the 2005 international conference on Computer Vision in Human-Computer Interaction
Multistream dynamic bayesian network for meeting segmentation

MLMI'04 Proceedings of the First international conference on Machine Learning for Multimodal Interaction
Probabilistic inference of gaze patterns and structure of multiparty conversations from head directions and utterances

JSAI'05 Proceedings of the 2005 international conference on New Frontiers in Artificial Intelligence
Multi channel sequence processing

Proceedings of the First international conference on Deterministic and Statistical Methods in Machine Learning
A multimodal analysis of floor control in meetings

MLMI'06 Proceedings of the Third international conference on Machine Learning for Multimodal Interaction
Writer identification for smart meeting room systems

DAS'06 Proceedings of the 7th international conference on Document Analysis Systems
Speaker diarization using low-cost wearable wireless sensors

Proceedings of the 3rd International Conference on Information and Communication Systems
The nonverbal structure of patient case discussions in multidisciplinary medical team meetings

ACM Transactions on Information Systems (TOIS)
Audio-visual fused online context analysis toward smart meeting room

UIC'07 Proceedings of the 4th international conference on Ubiquitous Intelligence and Computing
Multiple cameras for audio-visual speech recognition in an automotive environment

Computer Speech and Language
A gaze-based method for relating group involvement to individual engagement in multimodal multiparty dialogue

Proceedings of the 15th ACM on International conference on multimodal interaction
Towards developing a model for group involvement and individual engagement

Proceedings of the 15th ACM on International conference on multimodal interaction
Automatic identification of experts and performance prediction in the multimodal math data corpus through analysis of speech interaction

Proceedings of the 15th ACM on International conference on multimodal interaction
A Lattice-Computing ensemble for reasoning based on formal fusion of disparate data types, and an industrial dispensing application

Information Fusion
Human interaction categorization by using audio-visual cues

Machine Vision and Applications
Latent-Dynamic Conditional Random Fields for recognizing activities in smart homes

Journal of Ambient Intelligence and Smart Environments - Ambient and Smart Component Technologies for Human Centric Computing

Quantified Score

Hi-index	0.14

Visualization

Abstract

This paper investigates the recognition of group actions in meetings. A framework is employed in which group actions result from the interactions of the individual participants. The group actions are modeled using different HMM-based approaches, where the observations are provided by a set of audiovisual features monitoring the actions of individuals. Experiments demonstrate the importance of taking interactions into account in modeling the group actions. It is also shown that the visual modality contains useful information, even for predominantly audio-based events, motivating a multimodal approach to meeting analysis.