[HTCondor-users] Translating GPU device assignments?

[Michael Pelletier <Michael.V.Pelletier@xxxxxxxxxxxx>]

Hi folks,

I'm working on a GPU problem, and I'm trying to pin down an elegant way to inform the Caffe application which GPU it should use.

The HTCondor startd provides GPU_DEVICE_ORDINAL and CUDA_VISIBLE_DEVICES, and that's the argument that the "-gpu" command line option for Caffe needs. With another argument, "-fromenv," you can specify which command line arguments should be defined from an environment variable, using an EV named "FLAGS_gpu=1" for example, to set the equivalent of a command line argument of "-gpu=1."
, 
However, I'm not quite clear how to translate the HTCondor environment variable into the FLAGS_gpu, or to the command line arguments, in a graceful way.

Since there's nothing in the job ClassAd which provides this information, I can't use a $$() notation in the arguments line in the submit description.

I can't specify a post-match environment variable on the arguments line, directly.

I could submit a "/bin/sh -c" command line which could expand the environment variable as an argument via $(DOLLAR)CUDA_VISIBLE_DEVICES, but that gets rather messy, syntax-wise, in order to clump the arguments correctly..

I could do a wrapper to assign the environment variable, but that's just a variant of the sh -c approach.

Any clever suggestions? Is there a submit option I'm overlooking? Thanks!


Michael V. Pelletier
Principal Engineer
Information Technology
Future Technologies & Cloud
Integrated Defense Systems
Raytheon Company