Unsolved
This post is more than 5 years old
5 Practitioner
•
274.2K Posts
0
3594
Clip still exist after deleted for a while in API
Hello,
I would like to get adivce from Centera SDK/API experts.
My customer's environment should check whether if clip ID still exist in cluster once delete operation ran.
They are complaining about a delay of delete operation.
Once they run a command of deleting clips( like a 1000 clips ) in API, they check whether if deleted clips from previous operation exist - supposed to receive "none of them exist". But they received a result, saying around 20 clips still exist - approximately 2 percent.
This is happening every time they do delete fuction in API.
I have heard from customer that, their delete fuction actually integrate with checking existence operation. I have advised them to make a delay like 0.5 sec between delete/exist operation. However, customer found that all deleting clips removed clearly after 1 mins from first removal command.
Customer's centera environment is source, target and remote. they are connected as cascade replication.
I don't think this cascade replication design affects to deleting/checking existence function.
If any of you have a little idea of troubleshooting their issue, please help...
Thanks for reading this discussion.
Crystal
mfh2
208 Posts
1
July 24th, 2014 08:00
Hi Crystal -
This is a very interesting case. Before a clip.delete operation succeeds there is a 'reflection' CDF written to Centera which forces CentraStar to return 'no' to a subsequent clip.exists or clip.open request. This has been a pretty reliable mechanism for many years.
I think the only way to know what's truly going on is to have your customer record an SDK log of a test run that shows the problem. This can be accomplished by adding the environment variable FP_LOGPATH='\path\to\logfile.txt' to the execution environment. If you could then post the log to this thread some of our community experts could have a look.
Regards,
Mike Horgan
kim_marivoet
41 Posts
1
July 31st, 2014 01:00
To understand why this is happening, you need to understand a few details about how deletes work and how this relates to the 'exists' check (or reading back clips for that matter). As Mike indicated, when you delete a clip 2 things happen:
1- a reflection is written (some kind of anti-clip that tells Centera 'this clip is deleted')
2- the clip is physically deleted from the disks it's located on and from any table/db it's referenced in
If (1) succeeds, the delete is considered to be successful. (2) is allowed to fail or is best effort (suppose the clip is overprotected: the clip has more than 2 copies, and Centera doesn't 'see' all those copies). The expectation is that if (2) doesn't succeed or doesn't delete all copies or references, the reflection will block any exists check or read call.
This is the theory. In reality it might be, due to the nature of the Centera location mechanism (it's 'fuzzy' in some circumstances, meaning that it doesn't find back all locations, so it might find back the clip but miss the reflections) that for a small percentage you will still be able to read the clip for a short period after it has been deleted. After a while the location mechanism will 'repair' itself so the exists call will also 'see' the reflections, or the clip is cleaned up by one of the background tasks.
The likely-hood of this happening is increased when:
- your cluster is having health-issues (regens, db rebuilds, etc.)
- you are deleting a lot of clips with a lot of threads. This can result in internal queues overflowing so that (2) is not happening any more
At the end everything is fine, but it can indeed cause some confusion when you have deleted something and you can still read it back for a short period of time.
Regards,
Kim
mfh2
208 Posts
1
July 31st, 2014 04:00
Excellent explanation Kim, thank you.
I think there is still merit in examining an SDK log if the customer is still experiencing this issue, just to make sure the client isn't doing anything unusual.
Best Regards,
Mike Horgan