Taking the OXPath down the deep web

  • Authors:

  • Andrew Sellers;Tim Furche;Georg Gottlob;Giovanni Grasso;Christian Schallhart

  • Affiliations:

  • Oxford University, Wolfson Building, Oxford;Oxford University, Wolfson Building, Oxford;Oxford University, Wolfson Building, Oxford;Oxford University, Wolfson Building, Oxford;Oxford University, Wolfson Building, Oxford

  • Venue:
  • Proceedings of the 14th International Conference on Extending Database Technology
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

Although deep web analysis has been studied extensively, there is no succinct formalism to describe user interactions with AJAX-enabled web applications. Toward this end, we introduce OXPath as a superset of XPath 1.0. Beyond XPath, OXPath is able (1) to fill web forms and trigger DOM events, (2) to access dynamically computed CSS attributes, (3) to navigate between visible form fields, and (4) to mark relevant information for extraction. This way, OXPath expressions can closely simulate the human interaction relevant for navigation rather than rely exclusively on the HTML structure. Thus, they are quite resilient against technical changes. We demonstrate the expressiveness and practical efficacy of OXPath to tackle a group flight planning problem. We use the OXPath implementation and visual interface to access the popular, highly-scripted travel site Kayak. We show, how to formulate OXPath expressions to extract all booking information with just a few lines of code.