An Architecture for Fast Processing of Large Unstructured Data Sets

  • Authors:
  • Mark Franklin;Roger Chamberlain;Michael Henrichs;Berkley Shands;Jason White

  • Affiliations:
  • Washington University in St. Louis;Washington University in St. Louis/ Data Search Systems, Inc., St. Louis, MO;Data Search Systems, Inc., St. Louis, MO;Washington University in St. Louis;Data Search Systems, Inc., St. Louis, MO

  • Venue:
  • ICCD '04 Proceedings of the IEEE International Conference on Computer Design
  • Year:
  • 2004

Quantified Score

Hi-index 0.00

Visualization

Abstract

This paper presents a general system architecture tailored to performing searching, filtering, compression, encryption, and other operations on unstructured data streaming from a disk system. The system achieves high performance on such applications by providing for parallelism, hardware-application specialization and reconfiguration, and hardware placement near the disk systems. A limited prototype of a single compute node has been implemented and is described. The prototype is tailored to applications involving complex searching and its performance is compared to a pure software implementation having the same search capabilities. Performance is considered in terms of data set size, query string hit rate and query complexity. Performance results as a function of these parameters are presented and the results indicate that, for data set sizes above 1.4 MB, the prototype compute node is between one and two orders of magnitude faster than a pure software implementation. At high data set sizes, on an individual node, speedups of about 200 and a sustained throughput of 300 MB/sec have been achieved.