Boilerplate detection using shallow text features

  • Authors:

  • Christian Kohlschütter;Peter Fankhauser;Wolfgang Nejdl

  • Affiliations:

  • L3S Research Center / Leibniz Universität Hannover, Hannover, Germany;L3S Research Center / Leibniz Universität Hannover, Hannover, Germany;L3S Research Center / Leibniz Universität Hannover, Hannover, Germany

  • Venue:
  • Proceedings of the third ACM international conference on Web search and data mining
  • Year:
  • 2010

Quantified Score

Hi-index 0.00

Visualization

Abstract

In addition to the actual content Web pages consist of navigational elements, templates, and advertisements. This boilerplate text typically is not related to the main content, may deteriorate search precision and thus needs to be detected properly. In this paper, we analyze a small set of shallow text features for classifying the individual text elements in a Web page. We compare the approach to complex, state-of-the-art techniques and show that competitive accuracy can be achieved, at almost no cost. Moreover, we derive a simple and plausible stochastic model for describing the boilerplate creation process. With the help of our model, we also quantify the impact of boilerplate removal to retrieval performance and show significant improvements over the baseline. Finally, we extend the principled approach by straight-forward heuristics, achieving a remarkable detection accuracy.