Compression, Encryption and Dedupe - What's the deal??

Hello DPS world,

While delivering a tailored Networker / DD training today, a class participant asked me, What is the deal with sending compressed, encrypted data to Data Domain? Thought I would share the thoughts and discussion.

PowerProtect DD does variable block deduplication, meaning all files are segmented in to 4K-12K blocks and compared against existing blocks and only the unique ones are stored and duplicates are discarded. So maximum benefit can be achieved by sending the same data over and over to DD, which is what happens with backups mostly. Considering a 5-10% change rate, the rest is deduped on a daily / weekly basis. 

However when we present compressed / encrypted data to the DD's Dedupe algorithm, these two have a tendency to change the bit sequences in the blocks there by making it look like new blocks each time to PowerProtect DD and hence DD is forced to store the unique blocks, even though technically they belong to the same data, just jumbled in a different way. The end result? Dedupe suffers and we end up using more capacity on the DD.

So can we not compress or encrypt data backed up to DD? Of course we can. DD by default compresses all data that is backed up to it, and we could even choose between LZ, GZ and GZ fast algorithms to do that. PowerProtect DD also supports inline Data Encryption at rest and when enabled, the data is deduped first, then compressed, encrypted and written to disk, which is more effective and efficient both for performance and Capacity!!

The student liked the explanation! Hope you guys do too! Please post your questions below!

Guru