Investigating the statistical properties of user-generated documents

  • Authors:
  • Giacomo Inches;Mark James Carman;Fabio Crestani

  • Affiliations:
  • Faculty of Informatics, University of Lugano, Lugano, Switzerland;Faculty of Information Technology, Monash University, Melbourne, Australia;Faculty of Informatics, University of Lugano, Lugano, Switzerland

  • Venue:
  • FQAS'11 Proceedings of the 9th international conference on Flexible Query Answering Systems
  • Year:
  • 2011

Quantified Score

Hi-index 0.00

Visualization

Abstract

The importance of the Internet as a communication medium is reflected in the large amount of documents being generated every day by users of the different services that take place online. In this work we aim at analyzing the properties of these online user-generated documents for some of the established services over the Internet (Kongregate, Twitter, Myspace and Slashdot) and comparing them with a consolidated collection of standard information retrieval documents (from the Wall Street Journal, Associated Press and Financial Times, as part of the TREC ad-hoc collection). We investigate features such as document similarity, term burstiness, emoticons and Part-Of-Speech analysis, highlighting the applicability and limits of traditional content analysis and indexing techniques used in information retrieval to the new online user-generated documents.