[Condor-users] Condor feature suggestion: automatically compressed output files

[Alex Gontmakher <gsasha@xxxxxxxxxxxxxxxxx>]

Hi all,

Recently I find myself using Condor in the following way: I have a plain 
executable that produces large standard output, which I need stored for 
further analysis. In each execution, I need to run that executable under lots 
of different parameters, generating hundreds of jobs - and I would like to be 
able to store these outputs for further reference.

The problem is, the outputs take a lot of place, and I need to use bzip2 to 
compress them after they execute. However, I cannot just run bzip2 whenever I 
want, since some of the jobs can be still running, and so compressing the 
output file would result in broken output.

A solution would be to have Condor automatically compress the file contents if 
the output (and .err) file name ends with ".bz2" or ".gz". This should be 
easy to implement as bzip2 library is publicly available.

I think this feature would be useful not only to me, but to other users as 
well.

Regards,
Alex