Re: [Chtc-users] shared submit nodes and large portion of the CHTC pool are down

[Lauren Michael <lmichael@xxxxxxxx>]

Hello Again,

While power to the WID datacenter has been restored, we are performing diagnostic work and ensuring that all of our servers are working correctly before enabling user login to the submit nodes, submit-1.chtc.wisc.edu, and submit-3.chtc.wisc.edu.

We expect that functionality should be restored to the submit nodes by the end of the day, barring major complications, and will let users know when this has happened.

As always, please send an email to chtc@xxxxxxxxxxx if you have any other questions.

Best Wishes,

Your CHTC Team

---------------------------------

Lauren Michael

Research Computing Facilitator

On Tue, Aug 6, 2013 at 9:33 AM, Lauren Michael <lmichael@xxxxxxxx> wrote:

Hello Users,

With respect to the previous email:  While power was temporarily restored to the server room at the Wisconsin Institutes for Discovery last night, a more significant outage has subsequently occurred.

The outage is significant.  Our shared submit nodes (submit-1 and submit-3.chtc.wisc.edu) and many of the CHTC pool execute nodes are down as a result of this outage. We are told to expect significant delays for getting the CHTC pool and submit nodes back to normal.

At best, functionality may be restored by sometime today, but it could take longer. In the meantime, you will not be able to login to or submit jobs from our submit nodes, and pool productivity will be down (as the portion of the CHTC pool housed in WID is nonfunctional). We will send an email to all users when standard functionality has been restored and appreciate your patience with respect to this unforeseen issue. Please contact us by sending an email to chtc@xxxxxxxxxxx if you have any pressing questions.

Best Wishes,

The CHTC Team

  ---------- Forwarded message ----------

  From:*Aaron Moate <[3]moate@xxxxxxxxxxx>

  To:*[4]chtc-users@xxxxxxxxxxx

  Cc:*

  Date:*Mon, 05 Aug 2013 16:30:37 -0500

  Subject:*WID power outage

  CHTC Users,

  * *The WID datacenter has suffered a power outage. Power has

  been restored, and steps are being taken to restore normal

  operations. *Be aware that the execution of running jobs has

  most likely been impacted.

  Cheers,

  Aaron Moate

  CHTC Infrastructure Team

----- End forwarded message -----