Isilon Auto shutdown - Power cut (ups)

Hello Andreas213213,

Your APC is not able to initiate an auto shutdown of your Isilon.  Here is a link How to safely shut down an Isilon cluster prior to a scheduled power outage, which has some issues that can happen with an improper shutdown.  https://www.dell.com/support/kbdoc/en-us/000018989/isilon-onefs-how-to-safely-shut-down-an-isilon-cluster-prior-to-a-scheduled-power-outage

The point of shutting down initiated from the UPS is for unscheduled events like power or cooling fails or the server room has a flood.  Most large UPS's have sensors and are aware when events like this happen.  They have clients to install on various servers or provide dry contacts. 

It's nice to know how to cleanly shut down the Isilon if you know something is going to happen but there needs to be another way when something happens you don't know about ahead of time.

Could this be done from the UPS via IPMI?  Are there any clients that could be installed in OneFS?

Hello avhjr,

I am not aware of using ipmi and a UPS to do the shutdown. With an Isilon system you want to manage the system shutdown and not have an auto shutdown initiated.

Since it may not be possible to shut it down in the case of loss of power/cooling does it handle loss of power well?

The NVRAM battery should last between 3-5 days so that it can be powered back up after without any data loss. While not supported by us you may be able to run a script from APC Powerchute to run shutdown commands for the cluster. https://dell.to/3Ib1ymj When you restore power https://dell.to/3uJrpi3

My experience with the Eaton software is that it requires a client on the target machine.  The client talks to the UPS and knows when to initiate shutdown.  There are no clients for OneFS.

Perhaps a client on another machine could do a remote command to OneFS to shut it down?

Hello avhjr,

I have not personally seen this being able to be done on Isilon system.  Now that doesen’t mean that a customer hasn’t gotten it to work, I haven’t come across that customer.

I can understand that esp. if the NVRAM allows it to come back up without problems.  Our server room is small- four racks- and our UPS lasts for 10-15 min.  So we wouldn't be able to respond in time in the middle of the night.

You can do a fast shutdown manually from the GUI, which perhaps could be scripted via the CLI. however,  it is in the documentation that a rolling node reboot should have been done "recently" to ensure no errors on hardware e.g. PSU/HDD issues at boot. 

Hello all,

After initiating the command via script

isi config shutdown now we find it hard to power on the node.

It has blue light and is not completely shut. we have to keep pressed the power button to turn off and then again to power on.

We are trying via a bat file and pass it to apc powerchute as mentioned. 

The question here is that runing isi config shutdown on each node, is it safe ? why node are not turning off ?

We manage to automate the ssh connection from a workstation.

But we cannot pass through automatic the shutdown commands and what are those that will safely turn off the system. 

Any help will be much appreciated.

Thank you very much.

@avhjr​ This is our goal

@DELL-Sam L​ So,

#isi_for_array isi_flush

#isi config shutdown

Will shutdown properly the node ?

Can i flush the entire cluster and then shutdown all (shutdown the cluster) ?

Hello Andreas213213,

When you are shutting down a single node you need to specify which node you are shutting down in both your flush and shutdown commands. If you specify the node and do the flush first and wait about 10-15 minutes before issuing the shutdown command the node should shutdown.

If you are wanting to shutdown the entire cluster, then it is best to do it as outlined in this document. https://dell.to/4ayA3zF

Now you can flush the entire cluster and then shutdown however I am not sure of how long that might take to complete as it depends on the size of your cluster.