<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
Hi Joseph. </div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
You should add it to DOCKER_EXTRA_ARGUMENTS on the executer machine. </div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
If i recall correctly you should install some nvidia docker extension to have thia feature. </div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
One last thing use nvidia docker image as a base docker image. There is some environment variables allready in the image</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
<br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
I haven't done this for a long time but i have this working at my cluster. </div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
So it should be ok. </div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
<br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
David</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
<br>
</div>
<div style="color: rgb(33, 33, 33); background-color: rgb(255, 255, 255);" dir="auto">
<br>
</div>
<div id="ms-outlook-mobile-signature">Get <a href="https://aka.ms/AAb9ysg">Outlook for Android</a></div>
<div id="id-84f9427d-8cee-473f-9b21-74c0aa50acf3" class="ms-outlook-mobile-reference-message">
<div style="font-family: sans-serif; font-size: 15pt; color: rgb(0, 0, 0);"><br>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg"><strong>From:</strong> HTCondor-users <htcondor-users-bounces@cs.wisc.edu> on behalf of Fulem Josef <fulemj@btlnet.com><br>
<strong>Sent:</strong> Monday, November 1, 2021, 18:37<br>
<strong>To:</strong> htcondor-users@cs.wisc.edu<br>
<strong>Subject:</strong> [HTCondor-users] Docker universe + GPU (CUDA)+pytorch<br>
</div>
<br>
<div dir="ltr">Hello,
<div><br>
</div>
<div>Currently I'm trying to use the docker container with htcondor docker universe to run an application which requires the usage of GPU (CUDA) - pytorch. </div>
<div><br>
</div>
<div>When I do it via vanilla universe it works OK and the CUDA is available.</div>
<div><br>
</div>
<div>When I run this command: </div>
<div><i>condor_status -constraint '!isUndefined(DetectedGPUs)' -compact -af CUDADeviceName DetectedGPUs</i></div>
<div><br>
</div>
<div>then this is the output:</div>
<div><i>GeForce RTX 2070 SUPER GPU-d4decf4f, GPU-2a518ecd</i><br>
</div>
<div><i><br>
</i></div>
<div>Also, I have this in my htcondor config: </div>
<div><i>use feature : GPUs<br>
GPU_DISCOVERY_EXTRA = -extra</i><br>
</div>
<div><i><br>
</i></div>
<div>So it looks like the condor_gpu_discovery works OK.</div>
<div><br>
</div>
<div>When I build my docker image and I run it with <b style="font-style:italic">
--gpus all </b>or <i><b>--gpus device=0 </b></i></div>
<div>the CUDA is available and the application running in the container can use it.</div>
<div><br>
</div>
<div>But when I run it (the same docker image) via htcondor by using <b>docker universe
</b>the GPUs are not accessible even though the GPU is requested.</div>
<div><br>
</div>
<div>It looks like the docker run is missing the <i><b>--gpus</b></i> flag. Is it possible to pass this to the docker somehow?</div>
<div><br>
</div>
<div>Thank you very much for any suggestion or help.</div>
<div><br>
</div>
<div>Best Regards.</div>
<div><br>
</div>
<div>Josef </div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<br>
</div>
</body>
</html>