Synergistic use of direct manipulation and natural language
CHI '89 Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Integration and synchronization of input modes during multimodal human-computer interaction
Proceedings of the ACM SIGCHI Conference on Human factors in computing systems
Movement Phase in Signs and Co-Speech Gestures, and Their Transcriptions by Human Coders
Proceedings of the International Gesture Workshop on Gesture and Sign Language in Human-Computer Interaction
FG '00 Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition 2000
Histogram-based segmentation in a perceptually uniform color space
IEEE Transactions on Image Processing
Connecting language to the world
Artificial Intelligence - Special volume on connecting language to the world
Semiotic schemas: a framework for grounding language in action and perception
Artificial Intelligence - Special volume on connecting language to the world
Tracking dragon-hunters with language models
CIKM '06 Proceedings of the 15th ACM international conference on Information and knowledge management
Fusion of children's speech and 2D gestures when conversing with 3D characters
Signal Processing - Special section: Multimodal human-computer interfaces
Spontaneous speech understanding for robust multi-modal human-robot communication
COLING-ACL '06 Proceedings of the COLING/ACL on Main conference poster sessions
Connecting language to the world
Artificial Intelligence - Special volume on connecting language to the world
Semiotic schemas: A framework for grounding language in action and perception
Artificial Intelligence - Special volume on connecting language to the world
| Hi-index | 0.00 |
We describe a home lighting robot that uses directional spotlights to create complex lighting scenes. The robot senses its visual environment using a panoramic camera and attempts to maintain its target goal state by adjusting the positions and intensities of its lights. Users can communicate desired changes in the lighting environment through speech and gesture (e.g., "Make it brighter over there"). Information obtained from these two modalities are combined to form a goal, a desired change in the lighting of the scene. This goal is then incorporated into the system's target goal state. When the target goal state and the world are out of alignment, the system formulates a sensorimotor plan that acts on the world to return the system to homeostasis.