Re: [HTCondor-users] Docker nvidia runtime support

[JoÃo BaÃto <joao.bauto@xxxxxxxxxxxxxxxxxxxxxxx>]

Poorly explained by me...

What I was trying to say is that you could add an option (DOCKER_USE_NVIDIA_RUNTIME) that is checked when creating a docker container and, if set, it adds the runtime=nvidiaÂoption. The code would be similar to the one I posted with some changes.

    // add nvidia runtime if set

if (param_boolean("DOCKER_USE_NVIDIA_RUNTIME", true /*default*/,

true /*do_log*/, &machineAd, &jobAd)) {

runArgs.AppendArg("--runtime=nvidia");

}

The runtime requires docker >=1.12 so I'm not sure if the version check should be done by htcondorÂor aÂpriori by the administrator when installing docker-nvidia.ÂFrom some tests I performed, using the nvidia runtime by default doesn't have any impact on non-gpu containers.

Thanks,

JoÃo BaÃto

JoÃo BaÃto

---------------

ScientificÂComputing and Software Platform

Champalimaud Research
Champalimaud Center for the Unknown
Av. BrasÃlia, Doca de PedrouÃos
1400-038 Lisbon, Portugal
fchampalimaud.org

2018-07-24 22:34 GMT+01:00 John M Knoeller <johnkn@xxxxxxxxxxx>:

Sorry, I donât follow you.ÂÂ How does this change end up adding runtime=nvidia to the docker command line?

Â

-tj

Â

From: HTCondor-users <htcondor-users-bounces@cs.wisc.edu> On Behalf Of JoÃo BaÃto
Sent: Friday, July 20, 2018 4:43 AM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Docker nvidia runtime support

Â

Hi,

Â

We have been running HTCondor for a while mainly for Python/MATLAB workloads and we want to start packing our applications into container images however some of them depend on accessing NVIDIA GPUs.Â

Â

NVIDIA has released a container runtime for docker that allows direct access to the GPU without having to pass it to the container. Besides having to install this runtime, docker has to be called with --runtime=nvidia.Â

Â

We could allow users to run their jobs in a vanilla universe and call a job wrapper that eventually calls docker but this opens our servers to security vulnerabilities that we want to avoid. The docker universe already does everything we need in terms of restricting user permissions and taking care of mounting volumes automatically but lacks the possibility of passing additional arguments.

Â

Do you guys think it is possible or feasible to add this option to the docker universe?

Â

If I checked the source code correctly, something identical to this might work,

Â

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ // drop unneeded Linux capabilities

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ if (param_boolean("DOCKER_DROP_ALL_CAPABILITIES", true /*default*/,

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ true /*do_log*/, &machineAd, &jobAd)) {

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ runArgs.AppendArg("--cap-drop=all");

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ // --no-new-privileges flag appears in docker 1.11

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ if (DockerAPI::majorVersion > 1 ||

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ Â Â DockerAPI::minorVersion > 10) {

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ runArgs.AppendArg("--no-new-privileges");

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ }

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ }

Â

Â

Â

More info:Âhttps://github.com/NVIDIA/nvidia-docker

Â

Thanks!

Â

JoÃo BaÃto

---------------

ScientificÂComputing and Software Platform

Champalimaud Research
Champalimaud Center for the Unknown
Av. BrasÃlia, Doca de PedrouÃos
1400-038 Lisbon, Portugal
fchampalimaud.org

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/