Re: [HTCondor-users] Compress output/error/log files at the end of the job

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi David,

as long as you don't do any seeks in the output streams, maybe pipes
could work, i.e., create locally in each job/worker named pipes to write
the streams to - and then read and pipe the named pipe through zcat
compressed on to the shared file system.
A problem will probably, that the streams might already produce output
before a wrapper could set up pipes - maybe a preCmd could work?? (have
not tried it tbh)

Cheers,
  Thomas

On 18/04/2022 22.52, David Schultz wrote:
> Hi Stuart,
> 
> I do compressed stdout/err as part of my jobs on the execute side,
> partly to decrease network traffic through the schedd. To avoid
> malformed or incomplete log files, I use a wrapper script around my
> actual job payload which redirects the payload stdout/err to different
> files, compresses them at the end of the job, and transfers those files
> back explicitly. The only things in my HTCondor stdout/err are from my
> wrapper script for setting up the job environment.
> 
> I've tried several more clever things over the years, but have had
> issues with them. This is very simple and robust.
> 
> Best,
> David
> 
> 
> On Mon, Apr 18, 2022 at 1:57 PM Greg Thain <gthain@xxxxxxxxxxx
> <mailto:gthain@xxxxxxxxxxx>> wrote:
> 
>     On 4/18/22 05:03, Stuart Walker wrote:
>     > Hello,
>     >
>     > I have a slight problem that I wondered if someone could help me with.
>     >
>     > My error, log and output files are set like this in my job
>     submission script:
>     >
>     > output       =
>     /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/out/bdsim-$(Cluster)_$(Process).out
>     <http://desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/out/bdsim-$(Cluster)_$(Process).out>
>     > error       Â=
>     /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/error/bdsim-$(Cluster)_$(Process).error
>     <http://desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/error/bdsim-$(Cluster)_$(Process).error>
>     > log        Â=
>     /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-$(Cluster)_$(Process).log
>     <http://desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-$(Cluster)_$(Process).log>
> 
> 
>     Hi Stuart:
> 
>     Looking at the log file first, there shouldn't be too many bytes
>     written
>     to it per job, so I hope that's not a problem for your quota. HTCondor
>     does support safely sharing one big log file for many jobs -- perhaps
>     that would help with minimizing the number of files or inodes. You
>     could set
> 
>     log =
>     /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-some-workflow-name.log
>     <http://desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-some-workflow-name.log>
> 
>     Then all the jobs in some workflow that share that log file will safely
>     append to it.
> 
> 
>     Now, for the output and error files, I don't think we have a good
>     way in
>     HTCondor proper to compress them. If these jobs are part of a DAG,
>     then
>     a DAG postscript (which runs on the submit machine) could compress the
>     files. However, I suspect it is better overall to explicitly compress
>     them as part of the job on the execute side.
> 
> 
>     -greg
> 
>     _______________________________________________
>     HTCondor-users mailing list
>     To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx
>     <mailto:htcondor-users-request@xxxxxxxxxxx> with a
>     subject: Unsubscribe
>     You can also unsubscribe by visiting
>     https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>     <https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users>
> 
>     The archives can be found at:
>     https://lists.cs.wisc.edu/archive/htcondor-users/
>     <https://lists.cs.wisc.edu/archive/htcondor-users/>
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature