Re: [HTCondor-users] Running short-lived jobs on Condor

["Rowe, Thomas" <rowet@xxxxxxxxxx>]

The simulation replications I'm running can take anywhere between three days and thirty seconds. I have 80 slots on this network. Everything is great if runtimes are up towards a half hour. All slots are kept busy grinding away. But if I submit five hundred jobs that take 40 seconds each, I see at most about 30 slots put to use. The queue is actually often empty and all slots idle for one minute stretches except for the dagman job.

I played around with the NEGOTIATOR_INTERVAL setting, dropping it down to 20 seconds but that didn't seem to have too much impact.

What can I do to make it so that short running jobs don't result in a mostly idle cluster? There are many *_INTERVAL settings and it's not exactly obvious what knobs to turn. "HTCondor can't handle that case well" is a perfectly valid answer if that's the case.

Same question nine years ago without clear answers: https://lists.cs.wisc.edu/archive/htcondor-users/2006-September/msg00255.shtml