[Chtc-users] HTC Gluster back online

[chtc-users@xxxxxxxxxxx]

Greetings HTC Users,

Thank you to those of you who helped us take action today and for your patience. We were able to restore Gluster functionality after identifying the user whose jobs were quickly adding tens of TB to /mnt/gluster. Those with files in /mnt/gluster should check them to identify any potential file corruption or data loss that may have occurred when the Gluster filesystem was overloaded.

For the future:

Please always test and be aware of how much data your jobs will create before submitting a large batch of jobs. If you have been given access to /mnt/gluster on the HTC filesystem, always contact chtc@xxxxxxxxxxx before running any work that would require more than 1 TB of space in /mnt/gluster.

As always, if you have any questions or concerns, please don't hesitate to send an email directly to chtc@xxxxxxxxxxx!

Thank you for your ongoing cooperation,

Your CHTC Team

On Wed, Jun 29, 2016 at 10:39 AM:

Hello HTC Users,

(individuals only use our HPC Cluster can ignore this message)

Without permission from CHTC staff, someone has been adding significant data to /mnt/gluster in the CHTC's HTC System in the last day. The rate of data addition has completely filled the HTC Gluster, very quickly making it unusable for other jobs. Due to the scale of the problem, we are not yet able to figure out which user has caused the problem.

To all users with data in /mnt/gluster, PLEASE REMOVE AS MUCH DATA AS POSSIBLE AS SOON AS POSSIBLE.ÂAlso, please email us if it's possible if you have added more than 1 TB to /mnt/gluster in the last day.

Thank you,

Your CHTC Team

Lauren Michael -ÂResearch Computing Facilitator,ÂCenter for High Throughput Computing,ÂUniversity of Wisconsin - Madison

Other Affiliations:ÂAdvanced Computing Initiative;ÂWisconsin Institute for Discovery;ÂACI-REF

lmichael@xxxxxxxx,Âwww.tinyurl.com/LMichaelCalendar,ÂDiscovery 2262, 608.316.4430