Analysis of large data logs: an application of Poisson sampling on excite web queries

  • Authors:

  • H. Cenk Ozmutlu;Amanda Spink;Seda Ozmutlu

  • Affiliations:

  • Department of Industrial Engineering, Uludag University, Gorukle Kampusu, Bursa 16059, Turkey;School of Information Sciences and Technology, The Pennsylvania State University, 511 Rider I Building, 120 S. Burrowes Street, University Park, PA;Department of Industrial Engineering, Uludag University, Gorukle Kampusu, Bursa 16059, Turkey

  • Venue:
  • Information Processing and Management: an International Journal
  • Year:
  • 2002

Quantified Score

Hi-index 0.00

Visualization

Abstract

Search engines are the gateway for users to retrieve information from the Web. There is a crucial need for tools that allow effective analysis of search engine queries to provide a greater understanding of Web users' information seeking behavior. The objective of the study is to develop an effective strategy for the selection of samples from large-scale data sets. Millions of queries are submitted to Web search engines daily and new sampling techniques are required to bring these databases to a manageable size, while preserving the statistically representative characteristics of the entire data set. This paper reports results from a study using data logs from the Excite Web search engine. We use Poisson sampling to develop a sampling strategy, and show how sample sets selected by Poisson sampling statistically effectively represent the characteristics of the entire dataset. In addition, this paper discusses the use of Poisson sampling in continuous monitoring of stochastic processes, such as Web site dynamics.