The use of eye movements in human-computer interaction techniques: what you look at is what you get
ACM Transactions on Information Systems (TOIS) - Special issue on computer—human interaction
The logic of typed feature structures
The logic of typed feature structures
Communications of the ACM - Special issue on graphical user interfaces
Fundamentals of speech recognition
Fundamentals of speech recognition
Some computer science issues in ubiquitous computing
Communications of the ACM - Special issue on computer augmented environments: back to the real world
Intelligent multimedia interfaces
Intelligent multimedia interfaces
Productivity gains via an adaptive user interface: an empirical analysis
International Journal of Human-Computer Studies
The role of voice in human-machine communication
Voice communication between humans and machines
Bricks: laying the foundations for graspable user interfaces
CHI '95 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Recognizing Human Facial Expressions From Long Image Sequences Using Optical Flow
IEEE Transactions on Pattern Analysis and Machine Intelligence
Direct manipulation for comprehensible, predictable and controllable user interfaces
Proceedings of the 2nd international conference on Intelligent user interfaces
Human-computer interaction
Coding, Analysis, Interpretation, and Recognition of Facial Expressions
IEEE Transactions on Pattern Analysis and Machine Intelligence
Affective computing
QuickSet: multimodal interaction for distributed applications
MULTIMEDIA '97 Proceedings of the fifth ACM international conference on Multimedia
Tutorial 2: Designing and evaluating intelligent user interfaces
IUI '98 Proceedings of the 3rd international conference on Intelligent user interfaces
MVIEWS: multimodal tools for the video analyst
IUI '98 Proceedings of the 3rd international conference on Intelligent user interfaces
Defining a taxonomy of output modalities from an HCI perspective
Computer Standards & Interfaces
Visual Interpretation of Hand Gestures for Human-Computer Interaction: A Review
IEEE Transactions on Pattern Analysis and Machine Intelligence
Making systems sensitive to the user's time and working memory constraints
IUI '99 Proceedings of the 4th international conference on Intelligent user interfaces
Mutual disambiguation of recognition errors in a multimodel architecture
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
The visual analysis of human movement: a survey
Computer Vision and Image Understanding
Human motion analysis: a review
Computer Vision and Image Understanding
User modeling in adaptive interfaces
UM '99 Proceedings of the seventh international conference on User modeling
Ten myths of multimodal interaction
Communications of the ACM
Perceptual user interfaces (introduction)
Communications of the ACM
Perceptual user interfaces: multimodal interfaces that process what comes naturally
Communications of the ACM
Evaluation of eye gaze interaction
Proceedings of the SIGCHI conference on Human Factors in Computing Systems
Towards a fault-tolerant multi-agent system architecture
AGENTS '00 Proceedings of the fourth international conference on Autonomous agents
Automatic Analysis of Facial Expressions: The State of the Art
IEEE Transactions on Pattern Analysis and Machine Intelligence
The Recognition of Human Movement Using Temporal Templates
IEEE Transactions on Pattern Analysis and Machine Intelligence
A survey of computer vision-based human motion capture
Computer Vision and Image Understanding - Modeling people toward vision-based underatanding of a person's shape, appearance, and movement
Toward Machine Emotional Intelligence: Analysis of Affective Physiological State
IEEE Transactions on Pattern Analysis and Machine Intelligence - Graph Algorithms and Computer Vision
Detecting Faces in Images: A Survey
IEEE Transactions on Pattern Analysis and Machine Intelligence
Leonardo's Laptop: Human Needs and the New Computing Technologies
Leonardo's Laptop: Human Needs and the New Computing Technologies
The Psychology of Human-Computer Interaction
The Psychology of Human-Computer Interaction
Exploring Versus Exploiting when Learning User Models for Text Recommendation
User Modeling and User-Adapted Interaction
Modeling Multimodal Expression of User's Affective Subjective Experience
User Modeling and User-Adapted Interaction
Human Activity Recognition Using Multidimensional Indexing
IEEE Transactions on Pattern Analysis and Machine Intelligence
IEEE MultiMedia
Communications of the ACM
Communications of the ACM - Digital rights management
Multimodal 'eyes-free' interaction techniques for wearable devices
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Vision-Based Gesture Recognition: A Review
GW '99 Proceedings of the International Gesture Workshop on Gesture-Based Communication in Human-Computer Interaction
Hands on cooking: towards an attentive kitchen
CHI '03 Extended Abstracts on Human Factors in Computing Systems
Modeling humans in human-computer interaction
The human-computer interaction handbook
The human-computer interaction handbook
Face Image Retrieval Using HMMs
CBAIVL '99 Proceedings of the IEEE Workshop on Content-Based Access of Image and Video Libraries
CVPR '97 Proceedings of the 1997 Conference on Computer Vision and Pattern Recognition (CVPR '97)
“Put-that-there”: Voice and gesture at the graphics interface
SIGGRAPH '80 Proceedings of the 7th annual conference on Computer graphics and interactive techniques
A survey of research on context-aware homes
ACSW Frontiers '03 Proceedings of the Australasian information security workshop conference on ACSW frontiers 2003 - Volume 21
Evaluation of Tracking Methods for Human-Computer Interaction
WACV '02 Proceedings of the Sixth IEEE Workshop on Applications of Computer Vision
ICCV '95 Proceedings of the Fifth International Conference on Computer Vision
A unified approach to coding and interpreting face images
ICCV '95 Proceedings of the Fifth International Conference on Computer Vision
Techniques for Interactive Audience Participation
ICMI '02 Proceedings of the 4th IEEE International Conference on Multimodal Interfaces
The production and recognition of emotions in speech: features and algorithms
International Journal of Human-Computer Studies - Application of affective computing in human—Computer interaction
Information fusion in biometrics
Pattern Recognition Letters - Special issue: Audio- and video-based biometric person authentication (AVBPA 2001)
Automatic recognition of facial expressions using hidden markov models and estimation of expression intensity
Joint processing of audio-visual information for the recognition of emotional expressions in human-computer interaction
Eye Gaze Estimation from a Single Image of One Eye
ICCV '03 Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2
Tracking Articulated Body by Dynamic Markov Network
ICCV '03 Proceedings of the Ninth IEEE International Conference on Computer Vision - Volume 2
Facial expression recognition from video sequences: temporal and static modeling
Computer Vision and Image Understanding - Special issue on Face recognition
Face recognition: A literature survey
ACM Computing Surveys (CSUR)
Architecture and implementation of multimodal plug and play
Proceedings of the 5th international conference on Multimodal interfaces
A framework for rapid development of multimodal interfaces
Proceedings of the 5th international conference on Multimodal interfaces
Perceptual user interfaces using vision-based eye tracking
Proceedings of the 5th international conference on Multimodal interfaces
Tangible multimodal interfaces for safety-critical applications
Communications of the ACM - Multimodal interfaces that flex, adapt, and persist
Guidelines for multimodal user interface design
Communications of the ACM - Multimodal interfaces that flex, adapt, and persist
Computer vision in the interface
Communications of the ACM - Multimodal interfaces that flex, adapt, and persist
Robust clustering of eye movement recordings for quantification of visual interest
Proceedings of the 2004 symposium on Eye tracking research & applications
Experimental evaluation of vision and speech based multimodal interfaces
Proceedings of the 2001 workshop on Perceptive user interfaces
A robust algorithm for reading detection
Proceedings of the 2001 workshop on Perceptive user interfaces
Functional Unification Grammar: a formalism for machine translation
ACL '84 Proceedings of the 10th International Conference on Computational Linguistics and 22nd annual meeting on Association for Computational Linguistics
A cognitive meta-analysis of design approaches to interruptions in intelligent environments
CHI '04 Extended Abstracts on Human Factors in Computing Systems
A multimodal learning interface for grounding spoken language in sensory perceptions
ACM Transactions on Applied Perception (TAP)
Ontology and Taxonomy Collaborated Framework for Meeting Classification
ICPR '04 Proceedings of the Pattern Recognition, 17th International Conference on (ICPR'04) Volume 4 - Volume 04
Augmenting and sharing memory with eyeBlog
Proceedings of the the 1st ACM workshop on Continuous archival and retrieval of personal experiences
Multimodal detection of human interaction events in a nursing home environment
Proceedings of the 6th international conference on Multimodal interfaces
Bimodal HCI-related affect recognition
Proceedings of the 6th international conference on Multimodal interfaces
BT Technology Journal
IEEE Transactions on Pattern Analysis and Machine Intelligence
Using Eye Region Biometrics to Reveal Affective and Cognitive States
CVPRW '04 Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 5 - Volume 05
Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures
CVPRW '04 Proceedings of the 2004 Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04) Volume 10 - Volume 10
Automatic Analysis of Multimodal Group Actions in Meetings
IEEE Transactions on Pattern Analysis and Machine Intelligence
Real-Time Gesture Recognition by Learning and Selective Control of Visual Interest Points
IEEE Transactions on Pattern Analysis and Machine Intelligence
Automatic 2D Hand Tracking in Video Sequences
WACV-MOTION '05 Proceedings of the Seventh IEEE Workshops on Application of Computer Vision (WACV/MOTION'05) - Volume 1 - Volume 01
Conversing with the user based on eye-gaze patterns
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Individual differences in multimodal integration patterns: what are they and why do they exist?
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A living laboratory for the design and evaluation of ubiquitous computing technologies
CHI '05 Extended Abstracts on Human Factors in Computing Systems
Face and body gesture recognition for a vision-based multimodal analyzer
VIP '05 Proceedings of the Pan-Sydney area workshop on Visual information processing
Designing, playing, and performing with a vision-based mouth interface
NIME '03 Proceedings of the 2003 conference on New interfaces for musical expression
Machine Learning in Computer Vision (Computational Imaging and Vision)
Machine Learning in Computer Vision (Computational Imaging and Vision)
Real-Time Vision for Human-Computer Interaction
Real-Time Vision for Human-Computer Interaction
Affective multimodal human-computer interaction
Proceedings of the 13th annual ACM international conference on Multimedia
Human-Centered Multimedia: Culture, Deployment, and Access
IEEE MultiMedia
Meticulously Detailed Eye Region Model and Its Application to Analysis of Facial Images
IEEE Transactions on Pattern Analysis and Machine Intelligence
Evaluation of a haptic mixed reality system for interactions with a virtual control panel
Presence: Teleoperators and Virtual Environments - Special section: Legal, ethical, and policy issues associated with virtual environments and computer mediated reality
Emotion Recognition Based on Joint Visual and Audio Cues
ICPR '06 Proceedings of the 18th International Conference on Pattern Recognition - Volume 01
Human-centered computing: a multimedia perspective
MULTIMEDIA '06 Proceedings of the 14th annual ACM international conference on Multimedia
Rules of Play: Game Design Fundamentals
Rules of Play: Game Design Fundamentals
Ambient Intelligence: A Multimedia Perspective
IEEE MultiMedia
Live Soundscape Composition Based on Synthetic Emotions
IEEE MultiMedia
Audio-visual based emotion recognition-a new approach
CVPR'04 Proceedings of the 2004 IEEE computer society conference on Computer vision and pattern recognition
VACE multimodal meeting corpus
MLMI'05 Proceedings of the Second international conference on Machine Learning for Multimodal Interaction
Hotspot components for gesture-based interaction
INTERACT'05 Proceedings of the 2005 IFIP TC13 international conference on Human-Computer Interaction
Multimodal human computer interaction: a survey
ICCV'05 Proceedings of the 2005 international conference on Computer Vision in Human-Computer Interaction
Emotion analysis in man-machine interaction systems
MLMI'04 Proceedings of the First international conference on Machine Learning for Multimodal Interaction
Determining driver visual attention with one camera
IEEE Transactions on Intelligent Transportation Systems
Affective video content representation and modeling
IEEE Transactions on Multimedia
A survey on visual surveillance of object motion and behaviors
IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews
Human expression recognition from motion using a radial basis function network architecture
IEEE Transactions on Neural Networks
Content-based multimedia information retrieval: State of the art and challenges
ACM Transactions on Multimedia Computing, Communications, and Applications (TOMCCAP)
Personalized multimedia retrieval: the new trend?
Proceedings of the international workshop on Workshop on multimedia information retrieval
Human-centered multimedia systems: tutorial overview
Proceedings of the 15th international conference on Multimedia
Affective feedback: an investigation into the role of emotions in the information seeking process
Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval
Gesture Based Interface for Crime Scene Analysis: A Proposal
ICCSA '08 Proceedings of the international conference on Computational Science and Its Applications, Part II
HCC '08 Proceedings of the 3rd ACM international workshop on Human-centered computing
Voice enabling mobile financial services with multimodal transformation
International Journal of Mobile Communications
Ptz control with head tracking for video chat
CHI '09 Extended Abstracts on Human Factors in Computing Systems
Multimodal Interfaces: A Survey of Principles, Models and Frameworks
Human Machine Interaction
6th Sense--- Toward a Generic Framework for End-to-End Adaptive Wearable Augmented Reality
Human Machine Interaction
Generating 3D architectural models based on hand motion and gesture
Computers in Industry
Automatic facial expression recognition on a single 3D face by exploring shape deformation
MM '09 Proceedings of the 17th ACM international conference on Multimedia
Motion-path based gesture interaction with smart home services
MM '09 Proceedings of the 17th ACM international conference on Multimedia
Proceedings of the ACM International Conference on Image and Video Retrieval
Fusion engines for multimodal input: a survey
Proceedings of the 2009 international conference on Multimodal interfaces
Multimodal inference for driver-vehicle interaction
Proceedings of the 2009 international conference on Multimodal interfaces
Human-machine interaction issues in quality control based on online image classification
IEEE Transactions on Systems, Man, and Cybernetics, Part A: Systems and Humans
Context aware, multimodal, and semantic rendering engine
Proceedings of the 8th International Conference on Virtual Reality Continuum and its Applications in Industry
Applying Affect Recognition in Serious Games: The PlayMancer Project
MIG '09 Proceedings of the 2nd International Workshop on Motion in Games
Implicit human-centered tagging
ICME'09 Proceedings of the 2009 IEEE international conference on Multimedia and Expo
A software framework for multimodal human-computer interaction systems
SMC'09 Proceedings of the 2009 IEEE international conference on Systems, Man and Cybernetics
Multimodal interfaces: Challenges and perspectives
Journal of Ambient Intelligence and Smart Environments
A simple index for multimodal flexibility
Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Haptic and sound grid for enhanced positioning in a 3-D virtual environment
HAID'07 Proceedings of the 2nd international conference on Haptic and audio interaction design
User Modeling and User-Adapted Interaction
Affect- and behaviour-related assistance for families in the home environment
Proceedings of the 3rd International Conference on PErvasive Technologies Related to Assistive Environments
Facial expression recognition on multiple manifolds
Pattern Recognition
AdaBoost-based face detection for embedded systems
Computer Vision and Image Understanding
eMotion: un outil pour personnaliser la reconnaissance d'émotions
Proceedings of the Ergonomie et Informatique Avancee Conference
Cyclic and non-cyclic gesture spotting and classification in real-time applications
AMDO'10 Proceedings of the 6th international conference on Articulated motion and deformable objects
Key-press gestures recognition and interaction based on SEMG signals
International Conference on Multimodal Interfaces and the Workshop on Machine Learning for Multimodal Interaction
Personalization in multimedia retrieval: A survey
Multimedia Tools and Applications
Utilizing invariant descriptors for finger spelling American sign language using SVM
ISVC'10 Proceedings of the 6th international conference on Advances in visual computing - Volume Part I
Reconnaissance d'Emotions: un point de vue interaction multimodale
Conference Internationale Francophone sur I'Interaction Homme-Machine
Sketch recognition by fusion of temporal and image-based features
Pattern Recognition
Haptic and sound interface for shape rendering
Presence: Teleoperators and Virtual Environments
Display-camera calibration using eye reflections and geometry constraints
Computer Vision and Image Understanding
Information Processing and Management: an International Journal
Feel what you hear: haptic feedback as an accompaniment to mobile music playback
Proceedings of Interacting with Sound Workshop: Exploring Context-Aware, Local and Social Audio Applications
Vision-Based Fingertip-Writing Character Recognition
Journal of Signal Processing Systems
Head pose estimation in solving human-computer interaction problems
Pattern Recognition and Image Analysis
Interpreting dynamic meanings by integrating gesture and posture recognition system
ACCV'10 Proceedings of the 2010 international conference on Computer vision - Volume Part I
An innovative framework to support multimodal interaction with Smart Environments
Expert Systems with Applications: An International Journal
Humans and smart environments: a novel multimodal interaction approach
ICMI '11 Proceedings of the 13th international conference on multimodal interfaces
Fully automatic 3D facial expression recognition using a region-based approach
J-HGBU '11 Proceedings of the 2011 joint ACM workshop on Human gesture and behavior understanding
Voice activity detection and speaker localization using audiovisual cues
Pattern Recognition Letters
Affect recognition based on physiological changes during the watching of music videos
ACM Transactions on Interactive Intelligent Systems (TiiS) - Special Issue on Affective Interaction in Natural Environments
An adaptive solution for intra-operative gesture-based human-machine interaction
Proceedings of the 2012 ACM international conference on Intelligent User Interfaces
Draw your own story: paper and pencil interactive storytelling
ICEC'11 Proceedings of the 10th international conference on Entertainment Computing
A review on vision techniques applied to Human Behaviour Analysis for Ambient-Assisted Living
Expert Systems with Applications: An International Journal
Distributed multisensory signals acquisition and analysis in dyadic interactions
CHI '12 Extended Abstracts on Human Factors in Computing Systems
Number Matters: The Multimodality of Internet Use as an Indicator of the Digital Inequalities
Journal of Computer-Mediated Communication
Annotating traditional Chinese paintings for immersive virtual exhibition
Journal on Computing and Cultural Heritage (JOCCH)
On the simultaneous recognition of identity and expression from BU-3DFE datasets
Pattern Recognition Letters
Multimodal interfaces: Challenges and perspectives
Journal of Ambient Intelligence and Smart Environments
Nomadic gestures: A technique for reusing gesture commands for frequent ambient interactions
Journal of Ambient Intelligence and Smart Environments
Multimodal behavioral analysis for non-invasive stress detection
Expert Systems with Applications: An International Journal
Gestures used by intelligent wheelchair users
ICCHP'12 Proceedings of the 13th international conference on Computers Helping People with Special Needs - Volume Part II
Real-Time head pose estimation for mobile devices
IDEAL'12 Proceedings of the 13th international conference on Intelligent Data Engineering and Automated Learning
Multimodal human-machine interaction for service robots in home-care environments
SMIAE '12 Proceedings of the 1st Workshop on Speech and Multimodal Interaction in Assistive Environments
Speech and gesture interaction in an Ambient assisted living lab
SMIAE '12 Proceedings of the 1st Workshop on Speech and Multimodal Interaction in Assistive Environments
Proceedings of the 20th ACM international conference on Multimedia
KI'12 Proceedings of the 35th Annual German conference on Advances in Artificial Intelligence
Assessment of human head pose in human-computer interaction
Pattern Recognition and Image Analysis
Editorial: Advanced human-computer interaction
Computers in Human Behavior
Advanced Engineering Informatics
A survey of video datasets for human action and activity recognition
Computer Vision and Image Understanding
Beyond the basic emotions: what should affective computing compute?
CHI '13 Extended Abstracts on Human Factors in Computing Systems
Hand and foot gesture interaction for handheld devices
Proceedings of the 21st ACM international conference on Multimedia
Towards in situ affect detection in mobile devices: a multimodal approach
Proceedings of the 2013 Research in Adaptive and Convergent Systems
Multimodal smart interactive presentation system
HCI'13 Proceedings of the 15th international conference on Human-Computer Interaction: interaction modalities and techniques - Volume Part IV
Multimodal mathematical expressions recognition: case of speech and handwriting
HCI'13 Proceedings of the 15th international conference on Human-Computer Interaction: interaction modalities and techniques - Volume Part IV
Fusing multi-modal features for gesture recognition
Proceedings of the 15th ACM on International conference on multimodal interaction
Feature selection for gaze, pupillary, and EEG signals evoked in a 3D environment
Proceedings of the 6th workshop on Eye gaze in intelligent human machine interaction: gaze in multimodal interaction
Hierarchical fuzzy logic based approach for object tracking
Knowledge-Based Systems
ACM Transactions on Management Information Systems (TMIS) - Special Issue on Informatics for Smart Health and Wellbeing
Affective command-based control system integrating brain signals in commands control systems
Computers in Human Behavior
Review Article: Multimodal interaction: A review
Pattern Recognition Letters
ACM SIGAPP Applied Computing Review
Estimating and using absolute and relative viewing distance in interactive systems
Pervasive and Mobile Computing
Journal of Ambient Intelligence and Smart Environments
Context-aware HCI service selection
Mobile Information Systems - Advances in Network-Based Information Systems
Knowledge Elicitation Methods for Affect Modelling in Education
International Journal of Artificial Intelligence in Education
| Hi-index | 0.00 |
In this paper, we review the major approaches to multimodal human-computer interaction, giving an overview of the field from a computer vision perspective. In particular, we focus on body, gesture, gaze, and affective interaction (facial expression recognition and emotion in audio). We discuss user and task modeling, and multimodal fusion, highlighting challenges, open issues, and emerging applications for multimodal human-computer interaction (MMHCI) research.