Can we learn a template-independent wrapper for news article extraction from a single training site?

  • Authors:

  • Junfeng Wang;Chun Chen;Can Wang;Jian Pei;Jiajun Bu;Ziyu Guan;Wei Vivian Zhang

  • Affiliations:

  • Zhejiang Key Lab. of Service Robot, College of Computer Science, Zhejiang University, Hangzhou, China;Zhejiang Key Lab. of Service Robot, College of Computer Science, Zhejiang University, Hangzhou, China;Zhejiang Key Lab. of Service Robot, College of Computer Science, Zhejiang University, Hangzhou, China;School of Computer Science, Simon Fraser University, Vancouver, Canada;Zhejiang Key Lab. of Service Robot, College of Computer Science, Zhejiang University, Hangzhou, China;Zhejiang Key Lab. of Service Robot, College of Computer Science, Zhejiang University, Hangzhou, China;Microsoft Research, Redmond, WA, USA

  • Venue:
  • Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining
  • Year:
  • 2009

Quantified Score

Hi-index 0.00

Visualization

Abstract

Automatic news extraction from news pages is important in many Web applications such as news aggregation. However, the existing news extraction methods based on template-level wrapper induction have three serious limitations. First, the existing methods cannot correctly extract pages belonging to an unseen template. Second, it is costly to maintain up-to-date wrappers for a large amount of news websites, because any change of a template may invalidate the corresponding wrapper. Last, the existing methods can merely extract unformatted plain texts, and thus are not user friendly. In this paper, we tackle the problem of template-independent Web news extraction in a user-friendly way. We formalize Web news extraction as a machine learning problem and learn a template-independent wrapper using a very small number of labeled news pages from a single site. Novel features dedicated to news titles and bodies are developed. Correlations between news titles and news bodies are exploited. Our template-independent wrapper can extract news pages from different sites regardless of templates. Moreover, our approach can extract not only texts, but also images and animates within the news bodies and the extracted news articles are in the same visual style as in the original pages. In our experiments, a wrapper learned from 40 pages from a single news site achieved an accuracy of 98.1% on 3,973 news pages from 12 news sites.