[Chtc-users] 3rd Notice: HTC Gluster upgrade/downtime starting July 25 (MPI jobs affected); copy/remove all data ahead of time

[chtc-users@xxxxxxxxxxx]

Greetings,

This is another reminder that:

(1) Gluster-dependent jobs will no long *begin* running after some time tomorrow (July 21), and through the end of the necessary rebuild period. This will include HTC jobs that depend on MPI modules (which are supported from within the HTC Gluster filesystem). Jobs that are already running will continue to run until the HTC Gluster is taken down on July 25.

(2) There is still significant data in the HTC Gluster that MUST BE REMOVED before July 25 via transfer00.chtc.wisc.edu. It is not enough to only copy data off of Gluster to another location (which should still be done first). Users of the HTC Gluster need to completely delete this data, as any data left will extend the downtime necessary to rebuild a new HTC Gluster, perhaps by multiple days.

(3) Data in the HTC Gluster will not be backed up! If you do not copy data off of the HTC Gluster (AND remove it!), your data will be lost. We will not delay the downtime for users of the HTC Gluster who have waited too long to transfer their data to a non-CHTC location. This includes ALL files (including software) and even group directories in the /mnt/gluster location.

Full details are in our original email, further below. We will send a final reminder of the July 25 downtime (estimated to last up to a few days) on Monday, July 24.

Thank you, for your efforts in helping us make Gluster perform better for those who need it.

Your CHTC Team

On Fri, Jun 30, 2017 at 2:51 PM, <chtc-users@xxxxxxxxxxx> wrote:

Greetings HTC Gluster Users,

This email pertains only to users of theÂexternal Gluster file shareÂandÂMPI modulesÂon our HTC System. HPC Cluster users and HTC System users who do not use Gluster or MPI modules can ignore the below, and will not be affected.

We have recently completed internal tests to prepare for upgrading and rebuilding the HTC Gluster filesystem with better performance and user quotas (data space limits), which will promote better data practices by users to also improve performance. The upgrade will start on July 25 and will require deletion of all data in the HTC Gluster and we need all HTC Gluster users to take the below action prior to that time.

ALL HTC Gluster users need to remove data currently in the /mnt/gluster location ahead of July 25.

YOU NEED TO BEGIN REMOVING DATAÂNOW, AS WE WILL NOT DELAY FOR DATA TRANSFERS THAT HAVE NOT COMPLETED BY JULY 25.

• All data should be removed via theÂtransfer00.chtc.wisc.eduÂserver,Âconsistent withÂcurrent Gluster policies. Users seen transferring Gluster data in/out via a submit server will have their submit server accounts temporarily disabled, as doing so will cause performance issues.
• This includes data in group directories and all software installations.
• Any data left in /mnt/gluster by users will extend the upgrade process, making the HTC Gluster unavailable for a longer duration following July 25.Â
• We are not able to backup or transfer any user data to the new HTC Gluster. Instead, youÂwill be able to copy data necessary forÂnewÂwork into a new, empty user directory when the new HTC Gluster is available.
  
• The new HTC Gluster will have default starting quotas of 10GB and 100 files, which will be reflected in updatedÂGluster policiesÂcloser to July 25.

OVERALL UPGRADE TIMELINE

NOW-July 21: All user data needs to be copied off and removed from the /mnt/gluster location. Users observed transferring Gluster data through a submit server may have their submit server accounts temporarily disabled.

July 21:ÂNo new Gluster-dependent jobs will run in the HTC System until after the downtime (this includes MPI jobs). All other jobs will be unaffected.

July 25:ÂHTC Gluster will be inaccessible forÂ1-3 daysÂfor upgrade/rebuild.

after July 25:ÂUsers will be notified when the new HTC Gluster filesystem is available with empty user directories and default starting quotas.

As a reminder,ÂNO data from completed work should ever remain in GlusterÂand users should already have copies of important data elsewhere.ÂIn favor of optimizingÂfor better performance, theÂnew upgraded HTC Gluster will be more susceptible to unexpected data loss; as a result, it will be even more essential for users to keep copies of all data in a non-CHTC location.

Timeline reminders and any new details will be emailed as we approach July 25. As always, please contact us viaÂchtc@xxxxxxxxxxxÂwith any questions or comments, and do get started right away with removing data!!

Thank you, in advance, for your cooperation in helping us to upgrade and optimize the HTC Gluster for everyone who needs it!

Your CHTC Team

(care of Lauren Michael)

Lauren Michael -ÂResearch Computing Facilitator,ÂCenter for High Throughput Computing,ÂUniversity of Wisconsin - Madison

Other Affiliations:ÂAdvanced Computing Initiative;ÂWisconsin Institute for Discovery;ÂACI-REF

lmichael@xxxxxxxx,Âwww.tinyurl.com/LMichaelCalendar,ÂDiscovery 2262, 608.316.4430

_______________________________________________
Chtc-users mailing list
Chtc-users@xxxxxxxxxxx
To unsubscribe send an email to:
chtc@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/chtc-users