MelodicBrush: a novel system for cross-modal digital art creation linking calligraphy and music

  • Authors:

  • Michael Xuelin Huang;Will W. W. Tang;Kenneth W. K. Lo;C. K. Lau;Grace Ngai;Stephen Chan

  • Affiliations:

  • The Hong Kong Polytechnic University, Kowloon, Hong Kong;The Hong Kong Polytechnic University, Kowloon, Hong Kong;The Hong Kong Polytechnic University, Kowloon, Hong Kong;The Hong Kong Polytechnic University, Kowloon, Hong Kong;The Hong Kong Polytechnic University, Kowloon, Hong Kong;The Hong Kong Polytechnic University, Kowloon, Hong Kong

  • Venue:
  • Proceedings of the Designing Interactive Systems Conference
  • Year:
  • 2012

Quantified Score

Hi-index 0.00

Visualization

Abstract

MelodicBrush is a novel system that connects two ancient art forms: Chinese ink-brush calligraphy and Chinese music. Our system uses vision-based techniques to create a digitized ink-brush calligraphic writing surface with enhanced interaction functionalities. The music generation combines cross-modal stroke-note mapping and statistical language modeling techniques into a hybrid model that generates music as a real-time, auditory response and feedback to the user's calligraphic strokes. Our system is in fact a new cross-modal musical system that endows the ancient art of calligraphy writing with a novel auditory representation to provide the users with a natural and novel artistic experience. Experiment evaluations with real users suggest that MelodicBrush is intuitive and realistic, and can also be easily used to exercise creativity and support art generation.