Full backup from subdirectories backups

No - such will not work.

However you might want to look into 'synthetic fulls' - it will only run a true full for the first, the starting backup.

We were about to use synthethic full backups for the second and following full backups, but were specifically interested in avoiding the first truly full backup.

Thanks anyway.

Hi Noel,

would it be a solution for you to have two backup jobs:

1.) /data

2.) /data/incoming, /data/outgoing

For the /data backup job you define a directive to skip /data/incoming and /data/outgoing

Regards

Michael

It may be a workaround, but it is the complicated situation we want to eliminate in the long run.

Thanks

If I understand correctly, our DataDomain deduplicates the data only after they has arrived, so the bandwidth used would be exactly the same with or without DataDomain.

I agree with  Kleinenbroich.  

The end result is that you would have a complete backup of d:\data... although it would be split into 3 different savesets.

So if you want to perform recover the entire d:\data , then you may have to use saveset recovery of backups from d:\data, d:\data\incoming, and d:\data\outgoing

> If I understand correctly, our DataDomain deduplicates the data only after they has arrived, so the bandwidth used would be exactly the same with or without DataDomain.

If you configure the NetWorker client to use Client Direct and Data Domain Deduplication, then during backup the NetWorker client deduplicates the data before sending information to your Data Domain.  This function would reduce the amount of data sent to the DD, and thus lowering bandwidth usage.

Hi Wallace, Hi Noel,

Wallace is right, starting with (as far as I remember) NetWorker 8.0 the "client direct" feature was implemented also for NW client (if NW server has also 8.0 and client direct is enabled for the client). Using "client direct", the NetWorker (NW) client communicates direct with the DataDomain device. The NW client creates a hash for each block to be backed up and sends the hash to the DataDomain (DD) device. The single blocks of the backup are only transferred to the DD, if the hash is not yet stored on the DD. It's source based deduplication.

How well the deduplication works depend on a lot of parameters (e. g. encrypted files cannot be deduplicated).

Let's assume, that your files to be backed up could be deduplicated well and the system is configured to use client direct. In that case, you could seed the DD device with the hash values: You could try backup only a part of "d:\data" and after each backup cycle add additional files/ folder to the backup. If deduplication is working fine, the files from the last backup should run fast.

But please keep in mind: We're not in the backup business, we're in the recovery business: You will benefit from the deduplication during the backup, but not during the recovery. If you've to fulfill recovery SLAs, you might still have an issue.

Regrads

Michael

I ran this solution (it took several days to complete) in a set of folders and a direct superfolder of these.

The client has version 8.2.1.2.Build.764 and has Client Direct set up in its configuration. The server has version 8.2.0.3.Build.4.9.5 .

For the subfolders, speeds for first backup have been (KB/s):

For the superfolder, speed for first backup (after all subfolders have complete backups in the DataDomain) has been (KB/s):

so it seems DataDomain did not help at all with this superfolder case.

It does, indeed, work, as shown in the next full backups (KB/s):

This means that even if the block is already in the DataDomain, it is not identified as such since it belongs to "a different saveset".

Noel,

Data Domain doesn't distinguish between savesets. Deduplication is made in a different manner (take a look into Distributed Segment Processing).

So, I could be fairly sure your data domain is deduplicating because is made to do it. One thing you have to check is if your client is really using client direct. First, to know that, we should know how are the things connected in your environment (Data Domain connection to Storage Node vía FC or LAN, Client connection to Storage Node, Client Connection to Data Domain, etc).

But for now, you can check daemon.raw in your storage node when backup of the client is running, and you shoud see a message like " is using direct file save with Data Domain device ) what means that the client is using client direct and data already backed is not sent to data domain.

But a message like "Unable to perform direct file save with Data Domain device" is not good, and you should fix some things...

Hope it helps

Hi Noel,

I'm a bit confused about your last comments to the discussion: The backup performance of the second backup for the "next full backups" is (depending on the save set) between about 4x and 12x faster than the first backup.

Is there any other explanation for you than ddboost, explaining that the second backup is faster than the first one?

This is picture from an EMC document explaining the source based deduplication:

The hash value of the backup block is calculated, send to the DD device, if the hash is new (Unique), the block is compressed and send to the DD device. Otherwise the next hash is calculated and so on.

This process runs independent from the save set or the schedule (like full or incremental).

If you're using Linux as client I could provide you a test using tcpdump to prove it. Until you totally understand how ddboost works, it's a kind of miracle (e. g. ddboost does not use a fixed block size).

Regards

Michael

In the case for which the data is presented, both Client and Server are Windows machines, but I have no explanation for the performance on the second full backup for each saveset other that DataDomain being in place. On the other hand, I have found the messages ledugarte commented, and they indicate direct file save is not being used.

I'm puzzled.

Hi Noel,

the message "Unable to perform direct file save with Data Domain device" indicates, that the NetWorker client is not direct communicating with the DataDomain device, the client communicates via the NetWorker storage node. In that case the ddboost protocol is used between NetWorker storage node and the Data Domain device.

Is it possible, that the there is a bottleneck between Storage Node and DD device? This would explain the behavior.

Regards

Michael