Re: [Condor-devel] Avoiding redundant executables in the SPOOL

[Alan De Smet <adesmet@xxxxxxxxxxx>]

Matt Hope <matthew.hope@xxxxxxxxx> wrote:
> Does this benefit for being specific to executable only? (i.e. does it
> make implementing significantly easier or faster) compared to doing
> this for all input files that go to the spool.

The ickpt files are a special case already.  The other input
files, "the sandbox", are very messy.  The job can change them
(you can't change the ickpt file).  They can have their ownership
changed at various times.  We'd need to implement copy-on-write
and lots of other cleverness.  It's a good idea, but much too
large of a task for the immediate term.

> Also there may be some size of file where the additional overhead of
> any solution makes it not worth bothering, I suggest that should be
> configurable.

I'll think about it, but I'm not sure it's worth the added
complexity.  Calculating the hash scales linearly with file
length.  The remaining steps are all pretty trivial in processor
time.

-- 
Alan De Smet                              Condor Project Research
adesmet@xxxxxxxxxxx                http://www.cs.wisc.edu/condor/