From research to practice: experiences engineering a production metadata database for a scale out file system

  • Authors:
  • Charles Johnson;Kimberly Keeton;Charles B. Morrey;Craig A. N. Soules;Alistair Veitch;Stephen Bacon;Oskar Batuner;Marcelo Condotta;Hamilton Coutinho;Patrick J. Doyle;Rafael Eichelberger;Hugo Kiehl;Guilherme Magalhaes;James McEvoy;Padmanabhan Nagarajan;Patrick Osborne;Joaquim Souza;Andy Sparkes;Mike Spitzer;Sebastien Tandel;Lincoln Thomas;Sebastian Zangaro

  • Affiliations:
  • HP Labs;HP Labs;HP Labs;Natero;Google;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage;HP Storage

  • Venue:
  • FAST'14 Proceedings of the 12th USENIX conference on File and Storage Technologies
  • Year:
  • 2014

Quantified Score

Hi-index 0.00

Visualization

Abstract

HP's Store All with Express Query is a scalable commercial file archiving product that offers sophisticated file metadata management and search capabilities [3]. A new REST API enables fast, efficient searching to find all files that meet a given set of metadata criteria and the ability to tag files with custom metadata fields. The product brings together two significant systems: a scale out file system and a metadata database based on LazyBase [10]. In designing and building the combined product, we identified several real-world issues in using a pipelined database system in a distributed environment, and overcame several interesting design challenges that were not contemplated by the original research prototype. This paper highlights our experiences.