Discovery network outage affecting HTC System and HPC Cluster

[chtc-users@xxxxxxxxxxx]

For All CHTC Users:

Since roughly 2:15pm, today, there has been an unplanned network outage in the Discovery building, where the HPC Cluster and portions of the HTC System are located. Though we are not readily able to log into some servers to confirm the full extent of interruptions to CHTC services, the following (at least) will be affected:

• The HPC Cluster is still inaccessible for user login during the outage, but may continue to run jobs; however, jobs making connections outside the cluster (moving data to/from other campus or off-campus locations) will likely fail and will have to be resubmitted.
  
• Jobs running onÂsomeÂHTC execute servers will have been interrupted, though HTCondor will re-queue them (in the Idle state) and run them on another server when possible. Execute servers in the Discovery building will not be able to accept new jobs during the outage, and jobs already running on most HTC System servers will continue normally.

Generally. the HTC System submit servers and transfer servers are still accessible via SSH because they are not in the Discovery building, and new jobs can still be submitted to the HTC System.

There may be additional issues introduced when the Discovery network comes back online, and we do not yet have information on when that will happen. We'll provide an update after we've had a chance to understand additional impacts and/or any necessary user actions following the outage.

Thank you for your patience, and continue to let us know if you have questions or issues.Â

Your CHTC Team

Lauren Michael -ÂResearch Computing Facilitator,ÂCenter for High Throughput Computing,ÂUniversity of Wisconsin - Madison

Other Affiliations:ÂWisconsin Institute for Discovery;ÂOpen Science Grid;ÂCaRCC

lmichael@xxxxxxxx,Âtinyurl.com/LMichaelCalendar,ÂDiscovery 2262, (608)316-4430