[HTCondor-users] Condor job has trouble loading singularity

[Zhangqier Wang <wangzqe@xxxxxxx>]

Dear experts,

   I am currently using condor 8.8.15. The condor works fine on the regular sl7, sl6 singularity. But when I try the singularity:  /cvmfs/unpacked.cern.ch/registry.hub.docker.com/coffeateam/coffea-dask:latest 

  It has trouble loading it. We could run this singularity interactively on the worknote.  The same condor job also works if we replace with the sl7 singularity. 

  The singularity command that condor runs is: 

   exec /usr/bin/singularity exec --bind $LOCAL_DIR --bind /etc/condor --bind /etc/passwd --bind /cvmfs $SingularityImage "$@"​

  The script is simply "ls".  The last few lines of the --debug information is:

DEBUG   [U=189419,P=20885] sylogBuiltin()                Sourcing /.singularity.d/env/01-base.sh

DEBUG   [U=189419,P=20885] sylogBuiltin()                Sourcing /.singularity.d/env/10-docker2singularity.sh

DEBUG   [U=189419,P=20885] sylogBuiltin()                Sourcing /.singularity.d/env/90-environment.sh

DEBUG   [U=189419,P=20885] sylogBuiltin()                Sourcing /.singularity.d/env/95-apps.sh

DEBUG   [U=189419,P=20885] sylogBuiltin()                Sourcing /.singularity.d/env/99-base.sh

DEBUG   [U=189419,P=20885] sylogBuiltin()                Overriding LD_LIBRARY_PATH environment variable

DEBUG   [U=189419,P=20885] sylogBuiltin()                Sourcing /.singularity.d/env/99-runtimevars.sh

DEBUG   [U=189419,P=20885] sylogBuiltin()                Running action command exec

DEBUG   [U=189419,P=20850] PostStartProcess()            Post start process

/condor/execute/dir_20807/condor_exec.exe: line 9: ls: command not found

DEBUG   [U=189419,P=20850] Master()                      Child exited with exit status 127

​​  I am wondering why the condor job fails.

Sincerely,

Zhangqier Wang