Context-aware resemblance detection for data deduplication with neural network

As the prevalence of cloud storage increases, many individuals and companies prefer outsourcing their data for backup and management. However, this has led to a significant increase in redundancy, decreasing storage utilization and wasting network bandwidth. While conventional resemblance detection methods remove redundancy among similar data by comparing the features extracted from each chunk’s content. However, we observed that small changes between similar data chunks may cause false dissimilarity detection by conventional resemblance detection techniques. This is because features derived solely from the chunk content are highly susceptible to various modification patterns. Fortunately, we have discovered that two chunks are likely to be similar if their surrounding chunks are also similar, a concept we refer to as “chunk-context”. Therefore, we propose a novel chunk-context aware resemblance detection method, called CARD, which includes a network-based chunk-context aware model and an N-sub-chunk shingles-based initial feature extraction strategy. By leveraging the Neural network, it can discover the complex patterns between the chunk-context and chunk content itself. A high-level understanding of the contextual information with chunk content can be synthesized into the representation of a chunk. The primary difference compared with others is that our design can significantly improves the accuracy or efficiency of resemblance detection by considering the chunk-context with chunk content itself. Furthermore, we implemented a CARD prototype and conducted extensive experiments using real workload, demonstrating that CARD can detect up to 75. 03% more redundant data and accelerate the resemblance detection operations by 5. 6 × to 86. 7 × faster than state-of-the-art work.

Authors

• Xuming Ye
• Wenlong Tian University of South China
• Yaping Wan
• Ruixuan Li Huazhong University of Science and Technology
• Weijun Xiao Virginia Commonwealth University
• Zhiyong Xu Suffolk University

Keywords

• Cloud storage
• Chunk context-aware
• Date deduplication
• Resemblance detection
• Neural network