Is visual information useful for music communication?

  • Authors:
  • Rumi Hiraga;Nobuko Kato

  • Affiliations:
  • Tsukuba University of Technology, Tsukuba, Japan;Tsukuba University of Technology, Tsukuba, Japan

  • Venue:
  • MSIADU '09 Proceedings of the 1st ACM SIGMM international workshop on Media studies and implementations that help improving access to disabled users
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

We have conducted a series of experiments on recognizing emotions conveyed on the hearing of musical performances as a means of building a musical performance assistance system with which deaf and hard of hearing people could communicate through musical performance. Until recently, we had assumed that visual information provided simultaneously with musical performance was useful in helping the emotions emitted from music to be recognized, especially if the information conveyed the same emotions as those expressed in the musical performance. Currently we are not sure that this assumption is correct after having gone through experiments we conducted. In this paper, we describe the background of our plan to build a musical performance assistance system and review our past experiments by comparing the emotions recognized by recipients among several types of stimuli. Reviewing the results of our experiments made us unsure that multichannel information that supposedly helps deaf and hard of hearing people to recognize emotions actually works well. Based on a new question we pose about multichannel music information, we describe an experiment that compare the stimuli provided by music types only, music with video sequences, and video sequences only. The results we obtained do not clarify the question that we pose.