A visualized communication system using cross-media semantic association

  • Authors:

  • Xinming Zhang;Yang Liu;Chao Liang;Changsheng Xu

  • Affiliations:

  • National Labratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijng, China and China-Singapore Institute of Digital Media, Singapore;National Labratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijng, China and China-Singapore Institute of Digital Media, Singapore;National Labratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijng, China and China-Singapore Institute of Digital Media, Singapore;National Labratory of Pattern Recognition, Institute of Automation, Chinese Academy of Sciences, Beijng, China and China-Singapore Institute of Digital Media, Singapore

  • Venue:
  • MMM'11 Proceedings of the 17th international conference on Advances in multimedia modeling - Volume Part II
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Can you imagine that two people who have different native languages and cannot understand other's language are able to communicate with each other without professional interpreter? In this paper, a visualized communication system is designed to facilitate such people chatting with each other via visual information. Differing from the online instant message tools such as MSN, Google talk and ICQ, which are mostly based on textual information, the visualized communication system resorts to the vivid images which are relevant to the conversation context aside from text to jump the language obstacle. The multi-phase visual concept detection strategy is applied to associate the text with the corresponding web images. Then, a re-ranking algorithm attempts to return the most related and highest quality images at top positions. In addition, sentiment analysis is performed to help people understand the emotion of each other to further reduce the language obstacle. A number of daily conversation scenes are implemented in the experiments and the performance is evaluated by user study. The experimental results show that the visualized communication system is able to effectively help people with language obstacle to better understand each other.