Unsolved
1 Rookie
•
19 Posts
0
187
High latency after v3.6 upgrade
Hello,
On 29FEB we upgraded our 5000T from v3.2 to v3.6 and right thereafter we see a high spike in latency. The spike begins daily at 10:45am till around 1:50pm UTC. The baseline for the last two years is around 0.56ms (0.62 read/0.40 write) and hardly any spikes . The latency spikes to 14.64ms (9.43 read/24.74 write) which results in severe issues for our applications.
From the performance graph I see that it has a blank period whilst the upgrade takes place and when the graph starts showing data again the latency is already going steady upwards. This is happening daily since the upgrade and we can set our watches as it is that consistent.
No other changes are known to have been implemented.
A case has been raised with DELL however just interested if anyone else have a similar pattern. Four out of our 5 PowerStores are running on v3.6 however only this 5000T array shows the behavior described above.
apetersen
1 Rookie
1 Rookie
•
19 Posts
0
March 8th, 2024 14:19
From DELL Support:
The performance metrics show drastically increased CPU utilization following the upgrade to PowerStoreOS. This suggests the presence of a known software issue on PowerStoreOS 3.6.0 that leads to high latency and CPU utilization.
There are two main options to resolve this at the moment. A hotfix version is available - version PowerStoreOS 3.6.0.0-2213424 (Hotfix) - that can be applied to PowerStore systems with this issue. For this, we would work with our escalation engineering team to request permission to apply the hotfix version to this appliance. This may require us to gather even more detailed logs from the PowerStore to confirm our suspicions and eligibility for the fix.
Alternatively, a permanent fix for this issue will be included in PowerStore 3.6.1, currently scheduled for release next week (3/14/24). No further investigation will be needed to apply this - we can simply install it as soon as it becomes available.