Hello,
I run a HTCondor pool v8.8.10 and would like to enable user-defined Singularity images for submitted jobs, but when submitting test jobs I'm running into issues of the form:
FATAL: container creation failed: mount ->/user/path error: can't remount /user/path: no such file or directory
The Singularity image itself is fine, and executes without issues on the login or compute nodes; the error occurs only when submitting an HTCondor Singularity job.
I set up the Singularity compute nodes with the following configuration, based on the
Singularity Support docs:
# /etc/condor/config.d/70-singularity.conf
SINGULARITY_JOB = !isUndefined(TARGET.SingularityImage)
SINGULARITY_IMAGE_EXPR = TARGET.SingularityImage
SINGULARITY_TARGET_DIR = /srv
SINGULARITY_BIND_EXPR = "/cvmfs,/ligo/home/ligo.org:/ligo/home/ligo.org:rw,/localscratch:/localscratch:rw"
SINGULARITY_IS_SETUID = false
HAS_SINGULARITY = HasSingularity
STARTD_ATTRS = $(STARTD_ATTRS),HAS_SINGULARITY
A test submit file is:
#
/ligo/home/ligo.org/bryce.cousins/workflows/singularity_condor/test.sub
universe = vanilla
executable = /ligo/home/ligo.org/bryce.cousins/workflows/singularity_condor/containerInfo.sh
getenv = True
environment = "SINGULARITY_PWD=/ligo/home/ligo.org/bryce.cousins/git.ligo/gstlal/tacc"
+SingularityImage = "/ligo/home/ligo.org/bryce.cousins/workflows/singularity_condor/gstlal.simg"
error = $(cluster)-$(process).err
queue 1
Submitting this job leads to an error:
FATAL: container creation failed: mount ->/ligo/home/ligo.org/bryce.cousins/workflows/singularity_condor error: can't remount /ligo/home/ligo.org/bryce.cousins/workflows/singularity_condor: no such
file or directory
I'm not sure the root cause, since the `/ligo/home/ligo.org/` NFS directory is bound in the compute node config. Other changes I have tried that still cause the same FATAL error:
Is there some other configuration change (either in the .sub file or on the compute node) that would work?
Thank you for any guidance.
Bryce
----- Bryce Cousins
LIGO R&D Engineer
Penn State Institute for Computational
and Data Sciences
bfc5288@xxxxxxx 814-867-3035 |