Re: [HTCondor-users] Help needed understanding cpu core usage with cgroups

[Greg Thain <gthain@xxxxxxxxxxx>]

On 04/13/2015 01:37 PM, Roderick Johnstone wrote:
> Yes, the "foreground (Owner)" are non-condor jobs that are not using 
> cgroups. Our situation is that we use condor to soak up spare cycles 
> on peoples desktops as well as using dedicated computer servers and 
> this was a test of the former situation where I was simulating the 
> workstation owner running their own code outside of condor.
>
> So, just to double check I understand this, the cgroups is really 
> working to allocate the relative number of cpus between different 
> condor jobs running in different slots according to the request_cpus 
> criterion in the job submit file, regardless of the number of threads 
> running in each job.

Right.  To put a finer point on it, the HTcondor cgroup code works to 
share cpu resources, but only among processes within cgroups that have 
agreed to share cpus, whether or not those cgroups are managed by HTcondor.
> The actual number of cpus that a condor job might run on is not really 
> constrained by cgroups because non-condor (non-cgroups) jobs can 
> squeeze out the condor jobs and if there are no non-condor jobs, the 
> condor jobs can take over all available cpus.
>
> Is that right?

Yes.  The kernel gives us little control over other peoples' processes.
> > You could fix this by putting the foreground jobs into their own cgroup,
> > or running them as a condor job proper.
>
> Thats not really an option for us in the general case.

You could also affinity-lock the foreground processes.

Also, I wonder if improving the job's nice-ness would help the 
situation.  The knob JOB_RENICE_INCREMENT defaults to 10 in some 
versions of condor, you may want to set it to 0 if it isn't already, and 
try launching the condor_master with a negative niceness.

-greg


-greg