[HTCondor-users] Compress output/error/log files at the end of the job

[Stuart Walker <stuart.walker@xxxxxxx>]

Hello,

I have a slight problem that I wondered if someone could help me with.

My error, log and output files are set like this in my job submission script:

output              = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/out/bdsim-$(Cluster)_$(Process).out
error               = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/error/bdsim-$(Cluster)_$(Process).error
log                 = /afs/desy.de/user/s/stwalker/luxe-bb-bdsim-model/bdsim-models/beam-halo-model-2022-04-12/t20-halo-from-distr-file/output/log/bdsim-$(Cluster)_$(Process).log

However sadly for my simulations that I am running there are a lot of these files, which are also very big, and my quota is really quite small.  So I want to compress them at the end of each job.  How is this possible?  I tried compress_files but it does not work.  I can't compress at the end of the executable because it would presumably end in a malformed log file as the gzip could end up appended to by stdout/err/log.  So what is the solution here? 

Thank you for your help.

Kind regards,
Stuart Walker