[Chtc-users] IMPORTANT: All CHTC servers need immediate reboot TODAY!!!

[chtc-users@xxxxxxxxxxx]

ATTENTION CHTC Users!!

Due to very recent information on a critical vulnerability in the operating systems we use for CHTC compute servers,

ALL CHTC SERVERS NEED TO BE REBOOTED TODAY (see below)

For CHTC's HTC System (HTCondor Pool via submit nodes):

The process to reboot all servers has already begun, and will take place over the next 24 hours due to the large number of servers.

What HTC users can expect:

• Temporary delays in access to submit servers during their reboot (planned for early tomorrow).
• Interruption of running jobs as execute servers are automatedly rebooted over the next 24 hours. Interrupted jobs WILL continue to be tracked and will be re-run by HTCondor.
• Delays in the running of newly-submitted jobs until all reboots are complete.

For CHTC's HPC Cluster (via head node: aci-service-1.chtc.wisc.edu):

The HPC Cluster will be rebooted at 3pm today, and brought back ASAP after that point.

What HPC users can expect:

• loss of SSH access to cluster head nodes (aci-service-1/2) during the reboot.
• JOBS WILL BE LOST AND NEED TO BE REBOOTED, as SLURM cannot recover jobs upon reboot.

CHTC staff will send emails when the reboot processes have completedÂand compute system functionality is restored. The security vulnerability applies to all RedHat-based Linux operating systems, including the Scientific Linux operating system we use in CHTC. The security of your work is of utmost importance to CHTC, and this specific vulnerability requires immediate action.

The timing of the security vulnerability and CHTC-wide reboot are completely unrelated to the previously-described downtime for /mnt/gluster and high-memory servers in the HTC System that was necessary this morning. We apologize for any interruption to your CHTC research!

Thank you,

Your CHTC Team

(care of)

Lauren Michael - Research Computing Facilitator,ÂUniversity of Wisconsin - Madison

Center for High Throughput Computing; Advanced Computing Initiative; Wisconsin Institute for Discovery; ACI-REF

www.tinyurl.com/LMichaelCalendar, Discovery 2264, (608)316-4430