|
Unfortunately, no. HTCondor cannot enforce GPU memory limits on a process. In fact, it cannot even tell if a job is using more GPU memory than it requested. The GPU memory monitoring data we get from NVidia is per-GPU not per process.
In practice, if you duplicate GPUs with the -repeat or -divide arguments to condor_gpu_discovery, The jobs have to be well behaved and should not request more than a single GPU per job, or behave as if the total GPU memory that the request is all coming from
a single GPU.
If you duplicate GPUs, HTCondor cannot prevent jobs that share a GPU from interfering with each other, or even detect that it is happening.
-tj
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Ram Ban <ramban046@xxxxxxxxx>
Sent: Wednesday, March 5, 2025 12:30 PM
To: Jordan Corbett-Frank <jordan.Corbett-Frank@xxxxxxxxxx>
Cc: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] GPUs with htcondor
Hello All,
Are there any solutions to the above issues in any Htcondor version??
Thanks
Raman
I would love the feature to treat gpu ram similar to system ram and fit maximum jobs on a gpu.
My gpu ram across different jobs varies a lot, some require about less than 2GbB but some require around 10GB. I thought to divide X jobs per gpu and give proper num Gpus for jobs having higher requirement, but it is creating issues when there are multiple
GPUs on a machine.
2 major issues in this I have faced are -
1. How to put my job on hold if it is using more gpu memory than requested??(I have written custom logic for system ram using MemoryUsage variable on executor)
2. If I divide 2 jobs on each gpu each slot having ram 6GB(total gpu ram is 12GB), then my machine having 2 GPUs has 4 slots, Now first job comes requirment of 4GB and it is assigned on first slot and then another job comes with requirment of 10GB,
it gets 2 slots each of 1 gpu and some how it uses first gpu ram slot more and both my jobs get terminated due to gpu ram exceeded on first gpu. I am not able to handle this as well.
Thanks
Raman
Sorry to interject, but I'd like to second the request for partitionable slots to treat GPU memory the same way they do for system memory. We'd very much like to be able to pack as many GPU-dependent jobs as possible on the same machine. While we do manually
route gpu-dependent jobs to different nodes that are configured to allow a certain number of concurrent jobs to run on them, it's not as flexible as true partitionable slots.
Thank you,
Jordan
On 2/28/2025 11:48 AM, Ram Ban wrote:
Hello all,
I am currently using GPUs with Htcondor 10.2 version, Are there any new features related to GPUs in latest Htcondor version which will be helpful in better scheduling?
Hi Raman,
A few new items that come to mind since v10:
1. Job submit files now have various first-class entries for specifying what type of GPU your job needs (as opposed to embedding this info into expressions). E.g. your job submit file can now specify things like gpus_minimum_memory, gpus_minimum_runtime,
gpus_minimum_capability and gpus_maximum_capability.
2. GPUs and their properties will now be automatically discovered by default (no need to modify the configuration file on execution points).
3. GPUs that are not scheduled for use by a job are "hidden" from that job, even if the job clobbers/ignores environment variables like CUDA_VISIBLE_DEVICES.
4. Some helpful new commands, like "condor_submit -gpus" which shows servers with GPUs and the GPU properties
5. Improved management of GPUs accessed via Docker jobs.
6. Easier to configure an Execution Point (i.e. worker node) to prefer to run GPU jobs if any are waiting, but to "backfill" with non-GPU jobs if there are no idle GPU jobs waiting to run.
You can see a list of improvements at
https://htcondor.org/htcondor/release-highlights/
Also I am using partionable slots for gpu,cpu,ram, can I use gpu ram for that as well?
Not currently, but we have been thinking about this.... Are you saying you would like HTCondor to pack as many GPU jobs onto a single GPU device until the provisioned GPU memory is exhausted? Right now, if you know the GPU workload of your pool, what you
can do is configure Execution Points to run X jobs per GPU device.
regards,
Todd
--
Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing Department of Computer Sciences
|
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}