Networker is not deleting all expired savesets; DD is filling up

Hi,

I am having an issue where Networker is not deleting all expired savesets. We are running NW 19.9.0.2. I noticed that my DD is filling up fast so I decided to do some digging.

For your information, the retention time and browse time of our backups to our DD is 1 year.

After doing an sfs_dump on my DD, I noticed that there are many NW savesets files that are still present from November 2022 and older. This is not normal as our retention and browse time are 1 year. It is not all savesets of <= November 2022 that are still present in mminfo or sfs_dump. There are "only" 357 savesets  still present according to mminfo and I can correlate the dates of these savesets with files in sfs_dump. 

I did some digging using mminfo and nsrim -X -D9. Let's take this saveset as an exemple :

mminfo -q "ssid=1976935609" -r "ssid,volume,savetime,level,copies,cloneid,ssflags,ssbrowse,ssretent,clretent"
 ssid      volume           date     lvl copies  clone id ssflags browse  retent  clretent
1976935609 Daily.001       6/1/2020 incr     1 1591059641 vF     6/1/2021 6/1/2021 6/1/2021

Sorry for the formatting, but you may understand that this saveset has a browse time, retention time and clone retention time of June 1 2021. So it should be flagged as expired and be removed from the mediadb and DD by nsrim right ? This saveset is also an incremental, but I have many savesets that are full backups, if this matters.

I ran nsrim -X -D9 to try to have more details on this saveset structure . Here are the results of a search for this saveset in the output file of nsrim -X -D9 :

 Line 4984: 12/13/23 13:37:25.692967 ssid is 1976935609
 Line 4985: 12/13/23 13:37:25.692967 ssid 1976935609 is not an aborted ss
 Line 5044: 12/13/23 13:37:25.692967 hasdependents: searching ssid= 1976935609
 Line 5045: 12/13/23 13:37:25.692967 ssid '1976935609' is expired and protection_period is not set
 Line 5046: 12/13/23 13:37:25.692967 ssid '1976935609' is expired and protection_period is past
 Line 5068: 12/13/23 13:37:25.692967 hasdependents: searching ssid= 1976935609
 Line 5069: 12/13/23 13:37:25.692967 ssid '1976935609' is expired and protection_period is not set
 Line 5070: 12/13/23 13:37:25.692967 ssid '1976935609' is expired and protection_period is past
 Line 5093: 12/13/23 13:37:25.692967 hasdependents: searching ssid= 1976935609
 Line 5094: 12/13/23 13:37:25.692967 ssid '1976935609' is expired and protection_period is not set
 Line 5095: 12/13/23 13:37:25.692967 ssid '1976935609' is expired and protection_period is past
 Line 5114: 12/13/23 13:37:25.692967 hasdependents: searching ssid= 1976935609
 Line 5115: 12/13/23 13:37:25.692967 ssid '1976935609' is expired and protection_period is not set
 Line 5116: 12/13/23 13:37:25.692967 ssid '1976935609' is expired and protection_period is past
 Line 5135: 12/13/23 13:37:25.692967 Enter checkss: ssid = 1976935609, crosscheck = 1
 Line 5136: 12/13/23 13:37:25.692967 checkss: processing ssid = 1976935609, crosscheck = 1
 Line 5137: 12/13/23 13:37:25.692967 checkss_internal: crosscheck = TRUE for ssid = 1976935609
 Line 5138: 12/13/23 13:37:25.692967 crosscheckss: save set 1976935609 has cloneid 1591059641.
 Line 5156: 12/13/23 13:37:25.692967 ssid 1976935609 that has dependent(s), has insert=6/1/2020(1591059641), retent=6/1/2020(1670945882)
 Line 5174: 12/13/23 13:37:25.692967 ssid 1976935609 that has dependent(s), has insert=6/1/2020(1591059641), retent=6/1/2020(1670945882)
 Line 228471: 12/13/23 13:37:56.466928 looking for ssid 1976935609 browse: 31546758, retent 31546758 on Daily.001
 Line 228472: 12/13/23 13:37:56.466928 [is_orphaned_coverset:5363] Checking ssid 1976935609
 Line 228473: 12/13/23 13:37:56.466928 [is_orphaned_coverset:5367] Not a coverset 1976935609
 Line 228474: 12/13/23 13:37:56.466928 browsable 1976935609

I can see that the saveset is flagged as expired and there are no protection flag. So why doesn't it get deleted automatically ?

I do not understand this line : ssid 1976935609 that has dependent(s), has insert=6/1/2020(1591059641), retent=6/1/2020(1670945882)

If I look for any saveset with SSID 1591059641 and 1670945882 in mminfo, mminfo does not find any matches.

So this is where I am at. I have identified savesets that should be deleted from the mediadb and the DD, but it does not seem that NW wants to delete them. I would like to avoid having to delete them one by one, and possible "fix" why Networker is not deleting them automatically.

Also, as a side question, some of these savesets (out of the 357) have clones (to offsite tapes), so I must make sure that I delete only the original saveset located in the DD and not the clone saveset reference in the mediadb.

If anyone has any clues, that would be much appreciated.

Thank you,

Guillaume.