Re: [HTCondor-users] Compress output/error/log files at the end of the job

[David Schultz <david.schultz@xxxxxxxxxxxxxxxx>]

Hi Stuart,

I do compressed stdout/err as part of my jobs on the execute side, partly to decrease network traffic through the schedd. To avoid malformed or incomplete log files, I use a wrapper script around my actual job payload which redirects the payload stdout/err to different files, compresses them at the end of the job, and transfers those files back explicitly. The only things in my HTCondor stdout/err are from my wrapper script for setting up the job environment.

I've tried several more clever things over the years, but have had issues with them. This is very simple and robust.

Best,

David

On Mon, Apr 18, 2022 at 1:57 PM Greg Thain <gthain@xxxxxxxxxxx> wrote:

On 4/18/22 05:03, Stuart Walker wrote:
> Hello,
>
> I have a slight problem that I wondered if someone could help me with.
>
> My error, log and output files are set like this in my job submission script:
>
> output       = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/out/bdsim-$(Cluster)_$(Process).out
> error       Â= /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/error/bdsim-$(Cluster)_$(Process).error
> log        Â= /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-$(Cluster)_$(Process).log


Hi Stuart:

Looking at the log file first, there shouldn't be too many bytes written
to it per job, so I hope that's not a problem for your quota. HTCondor
does support safely sharing one big log file for many jobs -- perhaps
that would help with minimizing the number of files or inodes. You
could set

log =
/afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-some-workflow-name.log

Then all the jobs in some workflow that share that log file will safely
append to it.


Now, for the output and error files, I don't think we have a good way in
HTCondor proper to compress them. If these jobs are part of a DAG, then
a DAG postscript (which runs on the submit machine) could compress the
files. However, I suspect it is better overall to explicitly compress
them as part of the job on the execute side.


-greg

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/