User evaluation of the NASA technical report server recommendation service

  • Authors:
  • Michael L. Nelson;Johan Bollen;JoAnne R. Calhoun;Calvin E. Mackey

  • Affiliations:
  • Old Dominion University;Old Dominion University;NASA Langley Research Center;NASA Langley Research Center

  • Venue:
  • Proceedings of the 6th annual ACM international workshop on Web information and data management
  • Year:
  • 2004

Quantified Score

Hi-index 0.00

Visualization

Abstract

We present the user evaluation of two recommendation server methodologies implemented for the NASA Technical Report Server (NTRS). One methodology for generating recommendations uses log analysis to identify co-retrieval events on full-text documents. For comparison, we used the Vector Space Model (VSM) as the second methodology. We calculated cosine similarities and used the top 10 most similar documents (based on metadata) as "recommendations". We then ran an experiment with NASA Langley Research Center (LaRC) staff members to gather their feedback on which method produced the most "quality" recommendations. We found that in most cases VSM outperformed log analysis of co-retrievals. However, analyzing the data revealed the evaluations might have been structurally biased in favor of the VSM generated recommendations. We explore some possible methods for combining log analysis and VSM generated recommendations and suggest areas of future work.