Ongoing HTC Gluster file system outage over the weekend (2/29-3/1)

[chtc-users@xxxxxxxxxxx]

Hi everyone,

This message is for users of the high throughput (HTC) system who:

- use our Gluster file system to stage their data OR

- use MPI or licensed software modules on our HTC system

The Gluster file system is still experiencing periodic outages and will continue to do so over the weekend. HTC jobs that depend on Gluster for either data or module access may fail.

Next week, we will start transitioning from Gluster to a new data staging system. We will send more information about this to the chtc-users list next week. If your current work is being significantly impacted by not having consistent access to Gluster, feel free to reach out to us at chtc@xxxxxxxxxxx to discuss potential solutions.

Please continue to direct additional questions to chtc@xxxxxxxxxxx.

Cheers,
Your CHTC Team

---------- Forwarded message ---------
From: <chtc-users@xxxxxxxxxxx>
Date: Tue, Feb 25, 2020 at 5:09 PM
Subject: Ongoing outage of HTC Gluster file system
To: chtc-users <chtc-users@xxxxxxxxxxx>

Hi CHTC Users,

This message is for users of the high throughput (HTC) system who:

- use our Gluster file system to stage their data OR

- use MPI or licensed software modules on our HTC system

Portions of the Gluster file system have been failing intermittently since the weekend. Symptoms include specific files disappearing or error messages like "File not found" or "Transport endpoint not connected."

We have no indication that any data has been lost at this point, but jobs that depend on reading from, writing to, or using modules in Gluster will likely see failures due to the ongoing outage.

Jobs on the HTC system that do not depend on Gluster for either data staging or module use should not be impacted by this outage.

We will send a follow-up email to this address once we feel that we have restored Gluster to a more stable state where data access and module use will work consistently.

Please direct any additional questions to chtc@xxxxxxxxxxx.

Cheers,
Your CHTC Team

_______________________________________________
CHTC-users mailing list
CHTC-users@xxxxxxxxxxx
To unsubscribe send an email to:
chtc@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/chtc-users