This post is more than 5 years old
294 Posts
0
6314
New DPA install has high CPU/memory and does not trigger collection policies
We have an installation of DPA 6.1 that went live last Friday and has spent most of the time since then down! EMC's support hasn't been too helpful resolving this so I thought I would post it here and see if anyone can suggest anything.
Very early Saturday morning everything stopped collecting data and when I came in Monday morning CPU and memory on application server was on 100% use solidly (down to java.exe process), datastore was high but not maxed out; scheduled reports had failed to run and when I ran them manually there was no data since Friday night. Various combinations of stopping and starting services were tried, then rebooting application server. Logged case, support engineer WebExed in and turned off host monitoring on the DPA servers which I had configured on Friday, CPU dropped to 50%ish and memory to about 6GB. Next day, no data collected, CPU and memory solidly on 100%. Provided EMCReports and logs output for engineer which I have yet to hear back on.
I went through a process of turning collection policies off one-by-one hoping to establish what was causing the high load; will all policies and analyses off and services restarted CPU was on 50% and memory similarly about 4GB in use. Supposedly this system is doing nothing so a worrying baseline! Starting back collection policies one by one saw an initial spike to 100% for a few seconds then falling back; after everything was started back CPU/memory usage marginally different to without them on. Next day, no data. No collection policies had started; this may be my fault as there were collection policies under All Hosts and All Applications and I had only turned on the ones under All Applications for Networker; however turning the other ones on did nothing. Next I tried manually running the policies, they worked fine. I tried an historical data collection for the weekend and that worked. Still no policies running. Verified no overrides anywhere. Reset time between collections. Still nothing. Totally at a loss here. During this time I have also been monitoring resource usage on datastore, generally quite low, 10-20% CPU, that box is not being hammered.
So for now I have two issues:
(1) baseline resource usage is very high on application server
(2) collection policies do not run
Has anyone met anything like this before and been able to resolve it?
DavidHampson-rY
294 Posts
1
February 19th, 2014 03:00
We looked at it end of last week with a 4th line engineer, the problem was traced to one of the dashboard reports return an excessive amount of data and starving the system of resources. This was a new install of 6.1 and we had done nothing with the dashboard at all so all new installs may be at a similar risk. There is also a patch which Gareth mentions above that saw some improvement in CPU usage.
eagarg1
66 Posts
0
February 6th, 2014 18:00
Hi David,
I'm sorry to hear about the issues you're having getting DPA 6.1 running in your environment.
We have recently identified an issue with DPA 6.1 that sometimes causes high CPU usage and this can also result in issues with data collection. There will be a patch for DPA 6.1 to resolve this, and we expect this to be available by Monday, 10 Feb 2014 (and a new DPA 6.1 binary including this patch should be available for download from support.emc.com by 14 Feb).
I will try and contact you offline to get details on the SR number of the ticket you raised and to ensure we can get you the patch as soon as possible.
Regards,
Gareth
DavidHampson-rY
294 Posts
0
February 7th, 2014 02:00
Thanks Gareth, our SR is 60859304. I'd appreciate it if you could pass some information on this to your colleagues on this one.
eagarg1
66 Posts
0
February 7th, 2014 06:00
I had a look at the SR a little while ago and I see that support have made the patch available for you to download. This is expected to resolve the high CPU issue and with that resolved data collection should work correctly again.
Regards,
Gareth
DavidHampson-rY
294 Posts
0
February 10th, 2014 05:00
Patch applied; appeared to work at first but over the weekend high CPU came back, particularly on the application server. I stopped DPA agent and application services and there was an antivirus application using 50% CPU. I am assuming that there will be some files we will need to exclude from AV scanning, anyone have any ideas?
WestelCS
11 Posts
0
February 12th, 2014 14:00
Are you still stuck with the problem? I do not know much about this. But, I hope this thing helps somehow
Performance Management: Monitoring CPU Resources.
eagarg1
66 Posts
0
April 17th, 2014 10:00
Hi storagestudent.
Build 84695 is a recent patch, probably more recent than the patch referred to above. Have you logged a support ticket with EMC for the issues you're experiencing?
Regards,
Gareth
storagestudent
5 Posts
0
April 17th, 2014 10:00
Hello Gareth,
Specificaly for this issue i did not logged any SR, However there are
other SR's which are logged with SR SR62060550 for discovery issue and EMC
Support is aware of the issue.
I am not sure how can i disable dashboards or viewlets..
Thanks
Ashwani Dhakla
storagestudent
5 Posts
0
April 17th, 2014 10:00
DavidHampson
I am struggling to get DPA6.1 work normally since its deployment in my environment.
There have been multiple issues and 100% CPU Utilization is one of them, We are currently running on DPA-6.1.0.84695.
Can I Expect some favour from EMC or can you share patch download link to me.
eagarg1
66 Posts
1
April 17th, 2014 11:00
If you are logged in as an administrator then after closing the viewlets on each dashboard tab they should stay closed and that way be permanently disabled. In future when you login to DPA the dashboard section should be blank (no tabs opened).
eagarg1
66 Posts
0
April 17th, 2014 11:00
You need to look under APPLICATIONS in OBJECT LIBRARY and make sure that you have an entry there for Avamar with the name of your server under the PATH column. If it is not listed as an Avamar object under APPLICATIONS then something must have gone wrong when you discovered it.
If not listed then I'd suggest that you run the discovery wizard again.
If it is listed there, then go to INVENTORY / GROUP MANAGEMENT and click on OBJECTS NOT IN GROUPS. If the server is displayed on the right hand side you can drag and drop it in the CONFIGURATION / SERVERS / BACKUP SERVERS / EMC AVAMAR group (or any other group you want it in).
Regards,
Gareth
eagarg1
66 Posts
1
April 17th, 2014 11:00
To disable the dashboards, you need to open each dashboard tab (summary, backup, replication) and then close each panel (viewlet) on each of the tabs.
So if you go to DASHBOARDS in the DPA UI, you may have one or more dashboards displaying (summary, backup, etc). If not all dashboard tabs are open, click on the green PLUS + sign at the top of the dashboard window, click OPEN EXISTING, and make sure you open all dashboards.
Then go through each dashboard tab and close all the panels (viewlets). If you have a RecoverPoint Protected Capacity dashboard you may find that you cannot close all the panels/viewlets listed there but that is ok.
Regards,
Gareth
storagestudent
5 Posts
0
April 17th, 2014 11:00
Hello Gareth,
I have a host in object Library and its added in a group say avamar, But
when I try to run a report from Reports menu and asked to select scope, I
don't see this server under that group.
Can you please guide me how I can accomplish to get this host under Avamar
Group Scope and run a report for it.
Thanks
Ashwani
storagestudent
5 Posts
0
April 17th, 2014 11:00
Gareth,
I have note configured any of dashboards (summary, backup, replication).
These are default dashboards which are cause of CPU Usage Spikes.
I saw a significant drop in CPU Usage after closing these dashboards/
viewlets.
But They are opened again when I log into DPA. Is there any way out to
disable them permanently ?
Thanks
Ashwani Dhakla
Enterprise Computing Systems-Storage Systems
Desk:
Cell: +91-9818047149
storagestudent
5 Posts
0
April 17th, 2014 12:00
Thanks I was doing everything absolutely perfect.
I had to reopen the browser. Thanks for the support.
Thanks
Ashwani Dhakla