HPC Cluster Users: Remove ALL data ahead of Feb/Mar deletion; Reminder of head-node Policies

[chtc-users@xxxxxxxxxxx]

Greetings HPC Cluster Users,

(those who only use the HTC System can ignore the below)

We are writing for two reasons, see below:

1. Begin removing ALL data from the HPC Cluster

Users need to begin removing ALL data (from their /home/user space) on the HPC Cluster so that we can rebuild the HPC Cluster's filesystem during a downtime planned forÂlate February or early March of 2018.

• The entire HPC Cluster filesystem will need to be deleted for this downtime, and CHTC will not be able to keep any copies of user data to restore after the downtime.
• We will follow up in the coming weeks to elaborate on the downtime, and how users can expect to copy data (and software) back to the HPC Cluster after the downtime.
• TAKE ACTION NOW TO ENSURE THAT YOUR DATA EXISTS ELSEWHERE AND THAT ALL OLD DATA IS REMOVED FROM THE CLUSTER ASAP.

As a reminder ofÂCHTC data policies, which all users are responsible for:

• ONLY data that is being used or produced by actively-queued jobs should ever exist on the cluster.
• Data from completed work should be copied to another non-CHTC project location accessible to you, as soon as possible after jobs complete. Data left to accumulate reduces filesystem performance for yourself and all other users
• CHTC data locations are NOTÂbacked up, so you should always keep copies of essential data (scripts, submit files, etc.) in alternative, non-CHTC locations where you keep other research project data, and copy it back to the HPC Cluster when you need it, again.

In the coming weeks,Âwe will contact specific users who are obviously violating the above policiesÂby accumulating large amounts of data that have been left on the cluster for some time.

2. Reminder: Never run computational work on HPC Cluster head nodes

We have noticed a recent increase in the number of users running processes on the head nodes, which contributes to performance and filesystem issues/failures for ALL users.

As a reminder ofÂCHTC policies, which all users are responsible for:

• ALL computational work should ONLY be run within a SLURM-scheduledÂinteractive or batch job session, and never on the head nodes.ÂThis especially includes any scripts, software, or other processes that perform data manipulation/creation and long-running scripts for data management (including cron tasks).
• Only simple commands for file and directory management are appropriate to run on the head nodes (e.g. file transfers, compression/decompression of transferred data, directory creation, etc.).
• CHTC staff will deactivate the login access of users who violate the above policies, as compute-intensive tasks almost always create issues for other users by slowing or crashing the head nodes. We may not be able to immediately notify users that their accounts have been deactivated.

Repeat offenders may be required to involve their faculty sponsors to reinstate their login access, or may lose all access to CHTC resources. If you think youÂtrulyÂneed to run something on the head nodes that may violate the above policies, please don't hesitate to get in touch so that we can help explore practices that will not cause issues for other users.

Thank you, as always, for helping us to uphold and improve upon CHTC systems and their performance for all users. Please send any questions or concerns toÂchtc@xxxxxxxxxxx.

Regards,

Your CHTC Team