Re: [Condor-users] Condor feature suggestion: automatically compressed output files

[mjinks@xxxxxxxxxxxx]

On Thu, Jul 27, 2006 at 05:39:49PM +0300, Alex Gontmakher wrote:
> 
> Well, script has several problems (in addition to the hassle of having an 
> additional level of scripting in the jobs). If the files generated are very 
> large and highly redundant, it is possible for the uncompressed version to be 
> prohibitively large, while the compressed file size would be reasonable.
>
> In addition, our installation stores all the files over NFS - you can see that 
> compressing them after the fact adds significant overhead which would be 
> eliminated by compressing on the fly.

I'm very new to Condor but this doesn't strike me as the kind of
functionality that belongs in the job management system.

You've said your job writes to stdout -- if that's the case, wouldn't it
be trivial to just add "|bzip2 >outfile.bz2"?  No intermediate storage,
no NFS thrashing, no added code in an already complicated job control
system... And I don't think this would even be platform-specific, as
even Windows (IIRC) supports i/o redirection.

What am I missing?

--michael