Isilon CloudPools: Upgrading 8.x to 8.2.2.x or later

When upgrading a cluster using CloudPools, use the latest Isilon OneFS 8.2.2.0 installation bundle or the latest target version of OneFS 9.x or above

Pre-Upgrade Steps:
Review these steps and notes, as some of the preparation must be started at least three weeks before the upgrade.

When a cluster running CloudPools is upgraded from CloudPools v1 (pre-OneFS v8.2) to CloudPools v2 (OneFS v8.2 and later), files that are represented as SmartLink (stubbed) files on the cluster are converted between formats. The conversion process takes place after the upgrade is committed to the new version of OneFS. While this process does not block most operations, any Recall operations after conversion has started are placed into a Paused state until the conversion process completes, which can take a while.

After the Commit, CloudPools waits until it is safe to proceed with starting the CloudPools stub file conversion process. This event is called the 'Changeover' event and happens on all nodes simultaneously. It can take several minutes to complete.

Things that can slow down this Changeover process are Active SyncIQ syncs, NDMP backup sessions, and running CloudPools's archive or recall operations.

Pre-Upgrade Notes or Steps:

1. If CloudPools is used on this cluster, not all nodes on the network (NANON) must be corrected before the upgrade. NANON delays or blocks the ability to upgrade stub files in snapshots and causes the SnapshotDelete job to fail when trying to process file fill requests. This requires each node to have cloud storage connectivity. If this cannot be accomplished, reach out to Isilon Support for assistance before performing the upgrade.

2. If you are using SSLn or TLS and a certificate that required installation of a root or intermediate certificate previously, the certificate is not automatically migrated to the new certificate store on OneFS v8.2.2 or later. Make a note to perform this certificate import postupgrade. See KB article 491608.

3. CloudPools conversion doubles the number of inodes that are associated with archived files, this is translated as an increase in SSD usage. If SSD strategy is metadata or metadata-write and SSD usage before the upgrade is 60% or more, gather the COI dump "explained below" and open a case with support for assessment.

4. Any ongoing SyncIQ/NDMP jobs and any ongoing archive or recall cloud jobs, all should be completed before upgrading to the new OneFS. Otherwise, it delays the start of the conversion process "Changeover."

5. During the run of the conversion job, any recall jobs are paused per design, this does not mean that full inline access of files is impacted. "If recall of a file is necessary, a full inline-read can be done to copy the archived file to a new normal file copy and then replace the old archived copy with the new normal file copy."

6. On other hand, archive jobs during the conversion job work as per design.

Note:
A recall is a way to pull back all data from the cloud to storage permanently. This is not possible during the CloudPools conversion until completed. Inline-Access is still possible to read the file from the cloud during the conversion.

Gather COI dump:
Note: This step can take up to three weeks to complete. It is best to start this step as soon as possible.

This information is used to provide an estimate of the time to complete the conversion process. This command can take 1-3 days up to three weeks to complete, depending on the number of files stubbed to the cloud.

This data is also used in instances where upgrade issues have occurred.

The below command runs the dump command within a screen session, allowing the user to disconnect from the cluster while the script continues running:

# screen -dmS coi_dump bash -c 'isi_cpool_d_util -t coi > /ifs/data/Isilon_Support/$(date +%d%b)_coi_dump'

To verify if the coi_dump job has completed, run "screen -ls"/ If the coi_dump job has completed, the screen session no longer exists. Once the job is complete, run the following command to get the total number of files that must be processed (also used in Step 4).

# grep 'CMO ID:' /ifs/data/Isilon_Support/<date>_ coi_dump | wc -l

If the number of stub files that are reported is over ~50 Million, open a case with support. More details below in section "Upgrade: Step 6"

To calculate post OneFS upgrade CloudPools stub format conversion time, see KB 537705.

Note: Several items can delay the start of the conversion process. 

• New archive or recall tasks performed
• SIQ jobs that are in progress
• NDMP sessions in progress

 If any of these jobs or tasks are in progress, the stub conversion does not start until they complete or are canceled.


Upgrade
Step 1 
Ensure OneFS preupgrade steps have been completed.

For a cluster with SyncIQ policies:
In a SyncIQ environment, there is no restriction on performing and committing the OneFS upgrade on both Source and Target simultaneously. However, it is recommended that the OneFS upgrade be committed and the CloudPools conversion process started on the Target cluster before committing the OneFS upgrade on the Source cluster. This allows SyncIQ to send CP1.0 formatted stub files from the Source cluster to the Target cluster, where they are converted into CP2.0 formatted stub files.

If the OneFS upgrade is committed on the Source cluster first, SyncIQ operations fail until the OneFS upgrade is committed and the stub file conversion process started on the Target cluster. The only alternative to this is to reconfigure the SyncIQ policies to "Deep Copy" the CloudPools stub files. Deep Copy replicates the entire file to the target, rather than the stub, which takes longer, requires more space on the Target cluster, and is probably not wanted.

Note: SyncIQ operations and NDMP backups from a cluster may be slower than normal until all CloudPools stub files have been converted from CP1.0 to CP2.0 format.

Step 2
Perform the OneFS upgrade of the Isilon cluster. Depending on the number of nodes in your cluster, this may take some time.

Step 3
Proceed with the Commit of the cluster. This is usually a quick step, and there are no more reboots.

Step 4
As described earlier in this KB, after the Commit, CloudPools wait until it is safe to proceed with starting the CloudPools stub file conversion process. This event is called the Changeover event and happens on all nodes simultaneously. It can, however, take several minutes to complete.

Things that can slow down this Changeover are active SyncIQ sync jobs, active NDMP sessions, and CloudPools archives or recalls.

Changeover is the process of switching from CloudPools 1.0 to CloudPools 2.0 code paths and structures.

This process is relatively quick; within a few minutes after commit (especially when following this guide). The process includes:

• Waiting for completion of in-progress CloudPools tasks.
• Waiting for any NDMP backup and restore sessions involving CloudPools v1 stubs to complete.
• Waiting for any running SyncIQ jobs to complete.

# isi_for_array 'grep changeover /var/log/isi_cpool_d.log'

2019-03-11T12:18:00-04:00 <3.6> GaryB-1 isi_cpool_d[4729]: [0x814a15e10]:

/b/mnt/src/isilon/lib/isi_cpool_d/changeover_thread_pool.cpp:monitor_changeover_transition:56:

changeover monitoring thread exiting, shutdown pending? false changeover transition complete? Truethread exiting, shutdown pending? false changeover transition complete? True

Step 5
After the Changeover, CloudPools begin converting the existing CP1.0 stub files on the Target cluster to CP2.0 format.

The OneFS upgrade of the SyncIQ Source cluster can be committed when the Target cluster has started this CloudPools conversion process. Note: The CloudPools conversion process on both clusters, and SyncIQ, is quicker if you wait for the CloudPools conversion process to complete on the Target cluster before committing the OneFS upgrade to the Source cluster.

To monitor the progress of the CloudPools stub file conversion, run the following command:

# isi cloud jobs view 6 

Conversion is complete when the output shows Total Pending: 0 and Total Processing: 0.

Depending on the number and size of the CP1.0 files, this can take some time. See KB 537705 for assistance in determining an estimated time to completion.

Step 6
For a cluster with SyncIQ policies:
See the Release Notes for some of the known log spam that can happen during the conversion process.
This is partly because the Target cluster does not have write access to the Cloud Provider. Only a single cluster can have write access to the cloud provider.

Note: CloudPools stub file conversion does not start until all in-progress sync jobs and NDMP sessions have completed. These jobs can be canceled or allowed to complete before the conversion starts.

OneFS 8.2.2.0 allows sync policies to run while the conversion process is running. As a file is converted from CloudPools v1 to CloudPools v2, SyncIQ sees the conversion as a change and replicates the file to the target cluster as if the data has been updated. Since the SmartLink mappings that are updated are cluster local, not all these files must be replicated.

In most environments, this does not present any issues, and allowing SyncIQ to run during the conversion does not present any issues.

If a cluster has over ~ 50 million SmartLink files (see count that is taken above from COI dump), an optional procedure can be employed to allow SyncIQ to skip replication of the converted stub files where data has not been changed. Technical Support engagement is required to implement scripts and monitor this optional procedure. If this is wanted, reach out to Isilon Support for assistance.

Certificate Authority
If you are using SSL or TLS and a certificate that required installation of a root or intermediate certificate previously, the certificate is not automatically migrated to the new certificate store on OneFS v8.2.2 or later. The certificate can be imported now. See KB 491608.