[HTCondor-users] Summary: best practices for data untar/unzip

[Dimitri Maziuk <dmaziuk@xxxxxxxxxxxxx>]

On 11/01/2016 09:29 AM, Michael Pelletier wrote:
> This illustrates one of the problems of having started with UNIX in
about 1985 - these newfangled contraptions like ionice don't spring to
mind. Great idea, Thomas! On UNIX System III, we only had "nice" - and
we liked it!

(This reminds me: I should probably set elevator=deadline on all my
workers.)

I was wrong as it turns out: the tarball is actually mostly the same for
all jobs, the problem is these jobs can flock to other pools as far as
all the way to OSG.

There is an advantage to using the same submit file for all pools and to
not turning it into a twisty maze of ifs and cases.

Expecting a remote node to have a working ionice is a non-starter: IME
some OSG nodes appear to keep running without a working tar and/or gzip,
and even if installed, ionice only works with CFQ.

There are downsides and limitations to using caching solutions: squid
with bandwidth throttle etc., so the answer appears to be:

- we start using cgroups.

It's the right thing to do conceptually, anyway: it's not the job's
problem if the execute node can't handle the work. The submit side is
not the right place to deal with it, it's the node admin's job.

-- 
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu

Attachment: signature.asc
Description: OpenPGP digital signature