This post is more than 5 years old
4 Posts
0
1326
Deleting clips best practices questions
Hi,
I’m in charge of developing a system to delete all the clips that have ended their retention period. The estimated volume is at least 20TB of data to delete on a total of 65TB of data on our Centera.
In order to design the best system to automate the deletion of these clips, I would like to validate some points:
- As I understand, the only way to do this is to execute the delete command clip by clip. Am I right?
- Does the delete command is an expensive one in terms of Centera resource usage? For example, would you suggest that we do our mass deletion process in a period where there is less read access traffic?
- Intuitively, I would use a different profile to delete than the one used to write and read. Would you suggest that? What would be the benefits?
Thank you to enlighten me on this,
Estelle Charette
mfh2
208 Posts
0
October 19th, 2016 04:00
Hello Estelle -
I am assuming your application has a database that will allow you to identify the clip IDs that need to be deleted, so we can eliminate the question of how to find them. Addressing your questions above:
- Yes, you will need to call delete on each clip individually. Make sure you are able to handle the failure (and potential retry) of this call, in the case where a clip has already been deleted of the retention period has not expired (or legal hold has been applied, etc).
- The delete command is not particularly intensive, although it is slow. You may find you are limited to deleting a few thousand clips per hour. You can likely speed this up by running multiple delete operations in parallel, either using multiple threads or dividing up the work across multiple processes. You should probably have the storage administrator monitor the load on Centera during your testing and initial deployment just to be sure.
- You could certainly create a separate profile for the deletion operation. When you delete a clip on Centera a clip 'reflection' is created which records the time of the deletion and the profile that executed the command, so this could potentially be useful information in the future.
Be aware that when you perform mass deletes on Centera the free space on the cluster will not increase until after garbage collection runs (typically every 14 days). In fact, the free space may actually decline during the operation due to the creation of the reflections. This would be another operational aspect to be monitored while your utility is running.
Lastly, I would just like to point out that my company (Interlock Technology) is able to provide these type of custom Centera services on a fixed fee basis.
Regards,
Mike Horgan
CTO, Interlock Technology
www.interlock-tech.com
EstelleCharette
4 Posts
0
October 20th, 2016 10:00
Hi Mike,
Effectively, the clip IDs are kept in a database so it will be easy to identify which one to delete.
Thank you very much for your answer. It will help me evaluate the complexity of the solution needed to get through the 'cleaning' of the outdated data with the timeline desired by the user team.
Regards,
Estelle Charette