Interactive environments for music and multimedia

• Authors: Antonio Camurri;Pasqualino Ferrentino
• Venue: Multimedia Systems - Special issue on audio and multimedia
• Year: 1999

Multimodal Environments (MEs) are systems capable of establishing creative, multimodal user interactionby exhibiting real-time adaptive behaviour. In a typical scenario, one or more users are immersed in an environmentallowing them to communicate by means of full-body movement, singing or playing. Users get feedback from the environment in real time in terms of sound, music, visual media,and actuators, i.e. movement of semi-autonomous mobilesystems including mobile scenography, on-stage robots behaving as actors or players, possibly equipped with musicand multimedia output. MEs are therefore a sort of extension of augmented reality environments. From another viewpoint, an ME can be seen as a sort of prolongation of thehuman mind and senses. From an artificial intelligence perspective, an ME consists of a population of physical andas software agents capable of changing their reactions andtheir social interaction over time. For example, a gesture ofthe user(s) can mean different things in different situations,and can produce changes in the agents populating the ME.The paradigm adopted for movement recognition is that ofa human observer of the dance, where the focus of attentionchanges according to the evolution of the dance itself andof the music produced. MEs are therefore agents able to observe the user, extract "gesture gestalts", and change theirstate, including artificial emotions, over time. MEs open newniches of application, many still to be discovered, includingmusic, dance, theatre, interactive arts, entertainment, interactive exhibitions and museal installations, information atelier, edutainment, training, industrial applications and cognitive rehabilitation (e.g. for autism). The environment canbe a theatre, a museum, a discotheque, a school classroom,a rehabilitation centre for patients with a variety of sensory/motor and cognitive impairments, etc. The ME conceptgeneralizes the bio-feedback methods which already havefound widespread applications. The paper introduces MEs,then a flexible ME architecture, with a special focus on themodeling of the emotional component of the agents formingan ME. Description of four applications we recently developed, currently used in several real testbeds, conclude thepaper.