In a cluster used for the early development of the AN/SPY-6(V) radar system, the simulation software implemented its own internal lock-and-queue mechanism to share the GPU cards on the system, as it was back before HTCondor was able to provide GPU resource allocations. The CUDA_VISIBLE_DEVICES was unset, so each job had access to all GPUs, but since they didnât need 100% of a card, a queue was established in shared memory among all the running jobs, which managed time slots on one or the other of the M60 GPU cards.
Â
At one point I backported the GPU advertising pieces from a later HTCondor version while we were still working on testing and approvals for the HTCondor upgrade, but it turned out that assigning one job per card, or even multiple jobs per card by tweaking the ClassAd, left performance on the table as compared to the jobâs internal allocation mechanism across all available GPUs.
Â
I reckon with the astounding performance capabilities of the H100, this will become more of an issue for folks. The cost of the card makes each cycle left idle that much more costly, and I suspect it can be very challenging to write code capable of pushing such a card to 100% utilization without the right kind of problem to solve.
Â
Michael Pelletier
Principal Technologist
High Performance Computing
Classified Infrastructure Services
Â
C: +1 339.293.9149
michael.v.pelletier@xxxxxxx_______________________________________________Â
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Vikrant Aggarwal
Sent: Tuesday, February 27, 2024 6:51 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [External] Re: [HTCondor-users] Fractional GPUÂ
Hello Benedikt and David,
Â
Your comments are interesting.Â
Â
I have some queries on your responses:Â
Â
For Benedikt:
Â
Along with H100 which are supporting MIG
Â
We have recently introduced L40S which doesn't support MIG but has vGPU capabilities. I didn't get the chance to try out vGPU for your experience. Is condor_gpu_discovery able to detect each vGPUs as a separate GPU?Â
Â
For David:
Â
When you are saying we should control the number of processes, are we talking aboutÂMulti-Process Service (MPS) described hereÂhttps://docs.nvidia.com/deploy/mps/index.html?Â
Â
Thanks & Regards,
Vikrant Aggarwal
Â
Â
Â
On Fri, Feb 23, 2024 at 10:05âPM Dudu Handelman <duduhandelman@xxxxxxxxxxx> wrote:
Hi Larry.Â
I have done it before. how about not turning on htcondor gpu feature and use it as a normal server. For example, add a start statement that will only start jobs that specified +gpu=1. Now when the jobs start it will have access to all gpus. now you are in charge on how many processes will access each gpu.Â
Keep in mind you must control it.Â
Â
David
Â
Â
Â
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/