HPC cluster, certain HTC servers down (Oct 23)

[chtc-users@xxxxxxxxxxx]

Hi everyone,

We had a brief cooling outage in one of our server rooms this afternoon (Wed., Oct 23), resulting in many servers being automatically shut off. Impacts include:

HPC Cluster:

• All execute servers went down; all running jobs were interrupted.
  
• aci-service-1.chtc.wisc.edu went down.
• aci-service-2.chtc.wisc.edu is still up and /home directories are available.
  

HTC System:

• A portion of the HTC execute servers went down, including a subset of our servers that have GPUs; jobs running on those servers will have been interrupted and gone from "running" to "idle" in the queue.
  
• submit2.chtc.wisc.edu, submit3.chtc.wisc.edu and learn.chtc.wisc.edu went down; running jobs submitted from these three servers were likely interrupted.
• submit-1.chtc.wisc.edu, submit-4.chtc.wisc.edu, and most researcher-owned submit servers are still up.
  

On both systems:

• All jobs should still be present in the relevant queue; interrupted jobs should simply re-run once we start our execute servers.
  

We are currently starting the process of rebooting all affected servers. We will send another email to this address when this process is complete.

Please email our usual help address (chtc@xxxxxxxxxxx) with any questions or concerns!

Thanks,

Your CHTC team