Droplet: A Distributed Solution of Data Deduplication

• Authors: Yang Zhang;Yongwei Wu;Guangwen Yang
• Venue: GRID '12 Proceedings of the 2012 ACM/IEEE 13th International Conference on Grid Computing
• Year: 2012

Creating backup copies is the most commonly used technique to protect from data loss. In order to increase reliability, doing routinely backup is a best practice. Such backup activities will create multiple redundant data streams which is not economic to be directly stored on disk. Similarly, enterprise archival systems usually deal with redundant data, which needs to be stored for later accessing. Deduplication is an essential technique used under these situations, which could avoid storing identical data segments, and thus saves a significant portion of disk usage. Also, recent studies have shown that deduplication could also effectively reduce the disk space used to store virtual machine (VM) disk images. We present droplet, a distributed deduplication storage system that has been designed for high throughput and scalability. Droplet strips input data streams onto multiple storage nodes, thus limits number of stored data segments on each node and ensures the fingerprint index could be fitted into memory. The in-memory finger index avoids the disk bottleneck discussed in Data Domain, ChunkStash and provides excellent lookup performance. The buffering layer in droplet provides good write performance for small data segments. Compression on date segments reduces disk usage one step further.