Organizing news archives by near-duplicate copy detection in digital libraries

  • Authors:

  • Hung-Chi Chang;Jenq-Haur Wang

  • Affiliations:

  • Institute of Information Science, Academia Sinica, Taiwan;Department of Computer Science and Information Engineering, National Taipei University of Technology, Taiwan

  • Venue:
  • ICADL'07 Proceedings of the 10th international conference on Asian digital libraries: looking back 10 years and forging new frontiers
  • Year:
  • 2007

Quantified Score

Hi-index 0.00

Visualization

Abstract

There are huge numbers of documents in digital libraries. How to effectively organize these documents so that humans can easily browse or reference is a challenging task. Existing classification methods and chronological or geographical ordering only provide partial views of the news articles. The relationships among news articles might not be easily grasped. In this paper, we propose a near-duplicate copy detection approach to organizing news archives in digital libraries. Conventional copy detection methods use word-level features which could be time-consuming and not robust to term substitutions. In this paper, we propose a sentence-level statistics-based approach to detect near-duplicate documents, which is language independent, simple but effective. It's orthogonal to and can be used to complement word-based approaches. Also it's insensitive to actual page layout of articles. The experimental results showed the high efficiency and good accuracy of the proposed approach in detecting near-duplicates in news archives.