Gestural communication over video stream: supporting multimodal interaction for remote collaborative physical tasks

Authors:
Jiazhi Ou;Susan R. Fussell;Xilin Chen;Leslie D. Setlock;Jie Yang
Affiliations:
Carnegie Mellon University, Pittsburgh, PA;Carnegie Mellon University, Pittsburgh, PA;Carnegie Mellon University, Pittsburgh, PA;Carnegie Mellon University, Pittsburgh, PA;Carnegie Mellon University, Pittsburgh, PA
Venue:
Proceedings of the 5th international conference on Multimodal interfaces
Year:
2003

Citing 16
Cited 25

Findings from observational studies of collaborative work

International Journal of Man-Machine Studies - Computer-supported cooperative work and groupware. Part 1
Specifying gestures by example

Proceedings of the 18th annual conference on Computer graphics and interactive techniques
Integration of interpersonal space and shared workspace: ClearBoard design and experiments

ACM Transactions on Information Systems (TOIS)
GestureCam: a video communication system for sympathetic remote collaboration

CSCW '94 Proceedings of the 1994 ACM conference on Computer supported cooperative work
Analysis of gestures in face-to-face design teams provides guidance for how to use groupware in design

Proceedings of the 1st conference on Designing interactive systems: processes, practices, methods, & techniques
Collaboration in performance of physical tasks: effects on outcomes and communication

CSCW '96 Proceedings of the 1996 ACM conference on Computer supported cooperative work
Side-by-side collaboration: a case study

International Journal of Human-Computer Studies
Coordination of communication: effects of shared visual context on collaborative work

CSCW '00 Proceedings of the 2000 ACM conference on Computer supported cooperative work
GestureMan: a mobile robot that embodies a remote instructor's actions

CSCW '00 Proceedings of the 2000 ACM conference on Computer supported cooperative work
Improving interpretation of remote gestures with telepointer traces

CSCW '02 Proceedings of the 2002 ACM conference on Computer supported cooperative work
Effects of head-mounted and scene-oriented video systems on remote collaboration on physical tasks

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Assessing the value of a cursor pointing device for remote collaboration on physical tasks

CHI '03 Extended Abstracts on Human Factors in Computing Systems
On-Line Graphics Recognition

PG '02 Proceedings of the 10th Pacific Conference on Computer Graphics and Applications
DOVE: drawing over video environment

MULTIMEDIA '03 Proceedings of the eleventh ACM international conference on Multimedia
Sketch based interfaces: early processing for sketch understanding

Proceedings of the 2001 workshop on Perceptive user interfaces
Visual information as a conversational resource in collaborative physical tasks

Human-Computer Interaction

Effects of task properties, partner actions, and message content on eye gaze patterns in a collaborative task

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Analyzing and predicting focus of attention in remote collaborative tasks

ICMI '05 Proceedings of the 7th international conference on Multimodal interfaces
Comparing remote gesture technologies for supporting collaborative physical tasks

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
The effects of remote gesturing on distance instruction

CSCL '05 Proceedings of th 2005 conference on Computer support for collaborative learning: learning 2005: the next 10 years!
Combining audio and video to predict helpers' focus of attention in multiparty remote collaboration on physical tasks

Proceedings of the 8th international conference on Multimodal interfaces
Turn it this way: grounding collaborative action with remote gestures

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Ways of the hands

ECSCW'05 Proceedings of the ninth conference on European Conference on Computer Supported Cooperative Work
Representations can be good enough

ECSCW'05 Proceedings of the ninth conference on European Conference on Computer Supported Cooperative Work
Understanding the design space of referencing in collaborative augmented reality environments

GI '07 Proceedings of Graphics Interface 2007
Exploring interface with representation of gesture for remote collaboration

OZCHI '07 Proceedings of the 19th Australasian conference on Computer-Human Interaction: Entertaining User Interfaces
A review of telecollaboration technologies with respect to closely coupled collaboration

International Journal of Computer Applications in Technology
A collaborative guidance case study

AUIC '08 Proceedings of the ninth conference on Australasian user interface - Volume 76
Gestures over video streams to support remote collaboration on physical tasks

Human-Computer Interaction
Communicative Functions of Haptic Feedback

HAID '09 Proceedings of the 4th International Conference on Haptic and Audio Interaction Design
Gesture interaction in cooperation scenarios

CRIWG'09 Proceedings of the 15th international conference on Groupware: design, implementation, and use
Improving visibility of remote gestures in distributed tabletop collaboration

Proceedings of the ACM 2011 conference on Computer supported cooperative work
From ethnographic study to mixed reality: a remote collaborative troubleshooting system

Proceedings of the ACM 2011 conference on Computer supported cooperative work
Tertiary-Level Telehealth: A Media Space Application

Computer Supported Cooperative Work
A study of gestures in a video-mediated collaborative assembly task

Advances in Human-Computer Interaction
Auditory feedback in haptic collaborative interfaces

International Journal of Human-Computer Studies
Supporting hand gestures in mobile remote collaboration: a usability evaluation

BCS-HCI '11 Proceedings of the 25th BCS Conference on Human-Computer Interaction
Integrating the physical environment into mobile remote collaboration

MobileHCI '12 Proceedings of the 14th international conference on Human-computer interaction with mobile devices and services
Ownership and control of point of view in remote assistance

Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
A Haptic Tool for Group Work on Geometrical Concepts Engaging Blind and Sighted Pupils

ACM Transactions on Accessible Computing (TACCESS)
MobileHelper: remote guiding using smart mobile devices, hand gestures and augmented reality

SIGGRAPH Asia 2013 Symposium on Mobile Graphics and Interactive Applications

Quantified Score

Hi-index	0.00

Visualization

Abstract

We present a system integrating gesture and live video to support collaboration on physical tasks. The architecture combines network IP cameras, desktop PCs, and tablet PCs to allow a remote helper to draw on a video feed of a workspace as he/she provides task instructions. A gesture recognition component enables the system both to normalize freehand drawings to facilitate communication with remote partners and to use pen-based input as a camera control device. Results of a preliminary user study suggest that our gesture over video communication system enhances task performance over traditional video-only systems. Implications for the design of multimodal systems to support collaborative physical tasks are also discussed.