[HTCondor-users] Singularity and interactive jobs

[Alec Sheperd <alec.sheperd@xxxxxxxxxxxxxxxx>]

Hello,

We've recently run into two issues with Singularity and interactive 
jobs. We've been moving nodes to EL7 and running jobs inside of 
Singularity containers with EL6 to ease the process for users. One side 
affect has been that interactive jobs seem to not be working as I hope.

I've been able to confirm that what was done here worked on our system 
as well for running an interactive job inside a Singularity container.

https://lists.cs.wisc.edu/archive/htcondor-users/2017-August/msg00029.shtml. 


But I am unsure about the implications of specifying 
"UsePrivilegeSeparation no". I would think that would be fine, as were 
running sandbox directory Singularity images in the unprivileged user 
namespace. The main issue we are finding here is that we keep our 
sandbox directory images in a cvmfs repository, which in turn chowns 
everything to cvmfs:cvmfs, so it's able to properly jail the process in 
/var/empty/sshd/.

I had then wanted to switch to running interactive jobs outside of 
Singularity. That seems to have issues of it's own.

What I would like to do is specify where if "TARGET.InteractiveJob == 
true" to set "SINGULARITY_JOB = false".

In theory, I would think

"SINGULARITY_JOB = (TARGET.InteractiveJob is true) ? False : True"

But this doesn't seem to be the case.

The way that I currently define "SINGULARITY_JOB" allows the option to 
not run as a Singularity job, which I can confirm works outside of 
interactive jobs, if the "+NATIVE_OS = True" is defined in the submit file.

"SINGULARITY_JOB = TARGET.NATIVE_OS is True ? False : True"

However, even with condor_submit -interactive -a '+NATIVE_OS = True'

Is this behavior expected with interactive jobs? Or am I doing something 
wrong with my classads?

Thanks,

Alec