VxRail: VxRail Manager file-based backup not operational post upgrade

Scheduled VxRail Manager file-based backup is configured in the VxRail Manager crontab:

vx_mgr:/var/log # cat /etc/cron.d/vxm_schedule_backup
0 0 * * * root /usr/bin/logger 'VxM rotation backup start.' && python /mystic/vxm_backup_restore/vxm_backup_restore.py -b

There are no compatible backup files available since the upgrade completed:

vx_mgr:/var/log # mcp_python /mystic/vxm_backup_restore/vxm_backup_restore.py -l
Current user is root. We can do current job.
Starting to get authentication info.
Connecting to vCenter [vc_fqdn]
Fetched node_sn ['node1', 'node2', 'node3']
Download vxmbackup.json from datastore.
datacenter path is /VxRail-Datacenter
download file http_url: https://vc_fqdn/folder/VxRail_backup_folder/vxmbackup.json?dsName=VxRail-Virtual-SAN-Datastore-<cluster id>&dcPath=%2FVxRail-Datacenter
Found 3 VxRail backup item(s)
1) VxRailArchive_20241216125738_28492386.tgz(Incompatible: version not compatible)
2) VxRailArchive_20241216131518_28492386.tgz(Incompatible: version not compatible)
3) VxRailArchive_20241216133018_28492386.tgz(Incompatible: version not compatible)

The "messages" log shows the file-based backup starting at the scheduled timing:

2024-12-16T17:15:01.885958+00:00 vx_mgr CRON[19682]: (root) CMD (/usr/bin/logger 'VxM rotation backup start.' && python /mystic/vxm_backup_restore/vxm_backup_restore.py -b)
2024-12-16T17:15:01.932592+00:00 vx_mgr root: VxM rotation backup start.
2024-12-16T17:15:02.120986+00:00 vx_mgr CRON[19680]: (root) CMDEND (/usr/bin/logger 'VxM rotation backup start.' && python /mystic/vxm_backup_restore/vxm_backup_restore.py -b)

The "/var/log/mystic/VxRailBackupRestore.log" is 0 in size and is empty since the last upgrade:

vx_mgr:/home/mystic # ls -lh /var/log/mystic/VxRailBackupRestore.log
--w----r-T 1 root root 0 Dec 16 18:50 /var/log/mystic/VxRailBackupRestore.log

vx_mgr:/home/mystic # cat /var/log/mystic/VxRailBackupRestore.log

vx_mgr:/home/mystic #

Due to VxRail Manager internal Python changes, the backup script must be ran with "mcp_python" instead of "python." This is to load the correct Python environment packages in the newer releases.
File-based backups that are scheduled in a VxRail release prior to this change still reference "python" in the Operating System crontab.

When the old crontab entry is started, the script does not run correctly:

vx_mgr:/var/log # python /mystic/vxm_backup_restore/vxm_backup_restore.py -b
Usage of the script: mcp_python /mystic/vxm_backup_restore/vxm_backup_restore.py

This affects upgrades on both 7.0 and 8.0 VxRail releases:

• Code upgrades from before 7.0.520 to 7.0.520 or higher.
• Code upgrades from before 8.0.300 to 8.0.300 or higher.
• Code upgrades from before 7.0.520 to 8.0.300 or higher.

There is no fixed code available as this is further investigated.

Reconfigure the VxRail Manager file-based backup as root user using the command: "mcp_python /mystic/vxm_backup_restore/vxm_backup_restore.py -c"
With VxRail 8.0.370 and above the command "mcp_python vxm_backup_restore_limited_bandwidth.py -c" is used.

Example:

vx_mgr:/var/log # mcp_python /mystic/vxm_backup_restore/vxm_backup_restore.py -c
(8.0.370 and above:  mcp_python /mystic/vxm_backup_restore/vxm_backup_restore_limited_bandwidth.py -c )
Current user is root. We can do current job.
Download vxmbackup.json from datastore.
Starting to get authentication info.
Connecting to vCenter [vc_fqdn]
datacenter path is /VxRail-Datacenter
download file http_url: https://vc_fqdn/folder/VxRail_backup_folder/vxmbackup.json?dsName=VxRail-Virtual-SAN-Datastore-<cluster id>&dcPath=%2FVxRail-Datacenter
current backup rotation type is daily
Auto backup time is 0:0
Current auto backup will not keep logs
Current stored backup files limitation is [7]
rotation_type: [daily], do you want to change it? [y|n]n
Current schedule time is 0:0. Change it? [y|n]n
Current keeplog flag is 0. Do you want to change it? [y|n]n
Current rotation number is 7. Do you want to change it? [y|n]n
---------update crontab---------

0 0 * * * root /usr/bin/logger 'VxM rotation backup start.' && mcp_python /mystic/vxm_backup_restore/vxm_backup_restore.py -b

cronjob is updated.
datacenter path is /VxRail-Datacenter
download file http_url: https://vc_fqdn/folder/VxRail_backup_folder/vxmbackup.json?dsName=VxRail-Virtual-SAN-Datastore-<cluster id>&dcPath=%2FVxRail-Datacenter
schedule config is updated and uploaded to datastore
{"rotation":[{"id":0,"filename":"VxRailArchive_20241216125738_28492386.tgz","version":"7.0.483-28492386","node_number":3},{"id":1,"filename":"VxRailArchive_20241216131518_28492386.tgz","version":"7.0.483-28492386","node_number":3},{"id":2,"filename":"VxRailArchive_20241216133018_28492386.tgz","version":"7.0.483-28492386","node_number":3}],"backup_policy":{"rotation_type":"daily","week_day":"0","month_day":"1","backup_time_hour":"0","backup_time_minute":"0","backup_file_limit":"7","keep_log":"0"}}

current backup rotation type is daily
Auto backup time is 0:0
Current auto backup will not keep logs
Current stored backup files limitation is [7]
[Schedule config job END]

The crontab entry now references the correct command using "mcp_python":

vx_mgr:/var/log # cat /etc/cron.d/vxm_schedule_backup
0 0 * * * root /usr/bin/logger 'VxM rotation backup start.' && mcp_python /mystic/vxm_backup_restore/vxm_backup_restore.py -b

8.0.370 and above:
0 0 * * * root /usr/bin/logger 'VxM rotation backup start.' && mcp_python /mystic/vxm_backup_restore/vxm_backup_restore_limited_bandwidth.py -b

For additional information about the reference the relevant administration guide:

Document	Description
VxRail 8.0.x Administration Guide	Provide a general description of VxRail, how it works, and customer-facing administrative tasks.
VxRail 7.0.x Administration Guide