Re: [HTCondor-users] HTCondor/cgroups: limiting CPUs/pinning processes to CPUs with hyperthreaded CPUs

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi Michael and Bob,

many thanks for the info!
I have not considered CPU time sharing/known about its implementation in
HTCondor/cgroups. Using cpu shares should get a much better resource
utilization than pinning ;)

Cheers and thanks,
  Thomas


On 2016-02-04 16:30, Michael V Pelletier wrote:
> From: Thomas Hartmann <thomas.hartmann@xxxxxxx>
> Date: 02/04/2016 10:08 AM
>  
>> If a I understand the cgroup documentation correctly, a cgroup cannot be
>> limited to a "general number of cores" but can only be pinned to certain
>> cores. I.e., limiting the number of cores for a cgroup means to pin the
>> cgroup to as many dedicated cores on the system, or?
>> So, I guess the startd pins a job with core limit correspondingly in a
>> cgroup to cores, or?
>>
>> Is this actually a drawback, that processes cannot be switched between
>> cores (does the CPU would move anyway processes between cores?)?
>> How does actually a hyperthreaded system would look like - if a process
>> is pinned to "a hyperthreaded core", I guess the process would be moved
>> over the physical cores by the CPU, or?
> 
> With the default cgroup setup, the startd does not pin jobs to specific
> processors, but instead uses the cpu.shares functionality. The share
> assigned to a job is the number of requested cpus times 100, so a single
> core job gets 100, two-core gets 200, and so on.
> 
> This limit is only applied when there is contention for CPU time, however,
> so if a job wants to use 8 cores but only requested one, it can use eight
> only as long as there's idle capacity on other cores, but if the machine
> fills up it will be dialed back to its cpu.shares value of 100, on a single
> core.
> 
> See this page:
> https://oakbytes.wordpress.com/2012/09/02/cgroup-cpu-allocation-cpu-shares-examples/
> 
> 
> This has been important for compiled MATLAB jobs - unless the MATLAB code
> specifies a maximum compute thread count or has the singleCompThread
> command-line option, MATLAB will use all available cores, which is a
> bummer if your machine has a lot of cores and is also trying to run
> such a MATLAB job on each of them. The cpu.shares doesn't require a
> specific constraint in the MATLAB code, which means it will run full
> bore on the user's desktop, and full bore on an underutilized exec
> node, but won't step on everything else on a busy exec node.
> 
> If you do want affinity, for processor cache coherence considerations
> or the like, you can do that too, though.
> 
> There's a knob called "ENFORCE_CPU_AFFINITY" which causes each job and
> all its children to stay on a specific core, and "ASSIGN_CPU_AFFINITY"
> which enables affinity to work with dynamic slots and overrides the
> ENFORCE setting.
> 
>         -Michael Pelletier.
> _
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/
>

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature