[Chtc-users] SOAR update on submit.chtc.wisc.edu

[Bill Taylor <bt@xxxxxxxxxxx>]

 If you are not having your jobs automated by SOAR you
can ignore this message.

The situation today stemmed from the production volume of disk
filling up resulting in jobs not being able to complete. It was not 
possible
to empty old production space fast enough to keep all the current
jobs running.

Our cleaning script was removing runs more then 60 dags and
never removing results. Results were removed manually by
visual inspection of age as space was needed.

We will be doing several things to prevent this again. We will be 
looking to acquire
more production space and we will adjust the cleaning script to remove 
results and runs
older then 20 days.

If you had jobs running and I don't notify you, you do not
have to do anything as running jobs will be restarted when the
sweeps are turned back on.

If you have partial results which have not been packaged
please get me the project and run numbers and I will get to
it even while not here.

There will be a status email when SOAR production is restarted
later today or in the morning after the disk recovery in complete.

--
Bill Taylor 263-2656 (cell 219-4430)
Center for High Throughput Computing(CHTC)
Condor project
Computer Sciences