Re: [HTCondor-users] GPUs with htcondor

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

On 2/28/2025 11:48 AM, Ram Ban wrote:

Hello all,

I am currently using GPUs with Htcondor 10.2 version, Are there any new features related to GPUs in latest Htcondor version which will be helpful in better scheduling?

    Hi Raman,
    
    A few new items that come to mind since v10:
    
    1. Job submit files now have various first-class entries for
    specifying what type of GPU your job needs (as opposed to embedding
    this info into expressions).  E.g. your job submit file can now
    specify things like  gpus_minimum_memory,  gpus_minimum_runtime, 
    gpus_minimum_capability and gpus_maximum_capability.
    
    2. GPUs and their properties will now be automatically discovered by
    default (no need to modify the configuration file on execution
    points).
    
    3. GPUs that are not scheduled for use by a job are "hidden" from
    that job, even if the job clobbers/ignores environment variables
    like CUDA_VISIBLE_DEVICES.
    
    4. Some helpful new commands, like "condor_submit -gpus" which shows
    servers with GPUs and the GPU properties
    
    5. Improved management of GPUs accessed via Docker jobs.
    
    6. Easier to configure an Execution Point (i.e. worker node) to
    prefer to run GPU jobs if any are waiting, but to "backfill" with
    non-GPU jobs if there are no idle GPU jobs waiting to run.
    
    You can see a list of improvements at
    https://htcondor.org/htcondor/release-highlights/

Also I am using partionable slots for gpu,cpu,ram, can I use gpu ram for that as well?

    Not currently, but we have been thinking about this....  Are you
    saying you would like HTCondor to pack as many GPU jobs onto a
    single GPU device until the provisioned GPU memory is exhausted? 
    Right now, if you know the GPU workload of your pool, what you can
    do is configure Execution Points to run X jobs per GPU device.
    
    regards,
    Todd
-- 
Todd Tannenbaum <tannenba@xxxxxxxxxxx>  University of Wisconsin-Madison
Center for High Throughput Computing    Department of Computer Sciences