Speech pen: predictive handwriting based on ambient multimodal recognition

  • Authors:

  • Kazutaka Kurihara;Masataka Goto;Jun Ogata;Takeo Igarashi

  • Affiliations:

  • University of Tokyo;National Institute of Advanced Industrial Science and Technology, Japan;National Institute of Advanced Industrial Science and Technology, Japan;University of Tokyo

  • Venue:
  • Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
  • Year:
  • 2006

Quantified Score

Hi-index 0.01

Visualization

Abstract

It is tedious to handwrite long passages of text by hand. To make this process more efficient, we propose predictive handwriting that provides input predictions when the user writes by hand. A predictive handwriting system presents possible next words as a list and allows the user to select one to skip manual writing. Since it is not clear if people are willing to use prediction, we first run a user study to compare handwriting and selecting from the list. The result shows that, in Japanese, people prefer to select, especially when the expected performance gain from using selection is large. Based on these observations, we designed a multimodal input system, called speech-pen, that assists digital writing during lectures or presentations with background speech and handwriting recognition. The system recognizes speech and handwriting in the background and provides the instructor with predictions for further writing. The speech-pen system also allows the sharing of context information for predictions among the instructor and the audience; the result of the instructor's speech recognition is sent to the audience to support their own note-taking. Our preliminary study shows the effectiveness of this system and the implications for further improvements.