Graph-based bilingual sentence alignment from large scale web pages

  • Authors:
  • Yihe Zhu;Haofen Wang;Xixiu Ouyang;Yong Yu

  • Affiliations:
  • Apex Data & Knowledge Management Lab., Shanghai Jiao Tong University;Apex Data & Knowledge Management Lab., Shanghai Jiao Tong University;Apex Data & Knowledge Management Lab., Shanghai Jiao Tong University;Apex Data & Knowledge Management Lab., Shanghai Jiao Tong University

  • Venue:
  • NLDB'11 Proceedings of the 16th international conference on Natural language processing and information systems
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Sentence alignment is an enabling technology which extracts mass of bilingual corpora automatically from the vast and ever-growing Web pages. In this paper, we propose a novel graph-based sentence alignment approach. Compared with the existing approaches, ours is more resistant to the noise and structure diversity of Web pages by considering sentence structural features. We formulate sentence alignment to be a matching problem between nodes (bilingual sentences) of a bipartite graph. The maximum-weighted bipartite graph matching algorithm is first applied to sentence alignment for global optimal matching. Moreover, sentence merging and aligned sentence pattern detection are used to deal with the many-to-many matching issue and the low probability of aligned sentences with few mutual translated words issue respectively. We achieve good precision over 85% and recall over 80% on manually annotated data and 1 million aligned sentence pairs with over 82% accuracy are extracted from 0.8 million bilingual pages.