[HTCondor-users] Re: condor_ssh_to_job/interactive jobs with apptainer

[Joachim Meyer <jmeyer@xxxxxxxxxxxxxxxxxx>]

Hi again,

for reference, I managed to deal with a few things:

I realized I was using apptainer 1.3.4 from the (not up-to-date) PPA which is "pretty old" and went ahead and manually installed the 1.4.1 debs from the GitHub page.

With this, the GPUs seem to be usable and also correctly cgroup constrained as I'm used to from the Docker world (e.g. I only see exactly one).

However, e.g. nvidia-smi still isn't mounted - which feels like a bug - it just happens, as no PATH environment variable is set when launching apptainer.

They're looking up the binaries by inspecting the $PATH: https://github.com/apptainer/apptainer/blob/main/internal/pkg/util/paths/resolve.go#L149

In the StarterLog's "Env =" list before starting apptainer, I however don't see any PATH.

When using "getenv = PATH" in the submission file (e.g.), I get nvidia-smi mounted without problems (now I also see my PATH in the Env = line).

The remaining issues stay the same:

- condor_ssh_to_job leads to cgroup errors - which allows anything done here to escape the restrictions (e.g. I can see all GPUs with nvidia-smi here..) - I haven't found a difference here whether I used apptainer-suid or not.

- HOME is overwritten (I assume that is due to trying to workaround this stuff: https://htcondor.readthedocs.io/en/latest/faq/admins/interactive-containers-x11.html ? I really don't care about X11... :/ )

Any comments w.r.t the remaining issues are highly appreciated!

Cheers,

- Joachim

Am Mittwoch, 21. Mai 2025, 14:49:36 MitteleuropÃische Sommerzeit schrieb Joachim Meyer:

> Hi everyone,

>

> we're trying to enable Apptainer with our existing HTCondor install (+GPUs).

>

> I'm stuck at a few points:

> - condor_ssh_to_job leads to numerous warnings & errors with cgroups

> - when running a job in apptainer, GPUs are not being found (also nvidia-smi is not mounted) - this works if I just run the command from the "target user"

> - $HOME environment variable is (needlessly?) overwritten

> - do I actually want to use apptainer-suid? It sounds like it'd be more powerful if e.g. users have multiple groups - but haven't seen any mention of requiring this in HTCondor documentation.

> - are there any differences between container_image and manual +singularity_image shenanigans, except that container_image might use docker, if a docker:// uri is used?

>

> Version info on EP:

> $CondorVersion: 24.7.3 2025-04-22 BuildID: 803720 PackageID: 24.7.3-1+ubu22 GitSHA: e207c094 $

> $CondorPlatform: X86_64-Ubuntu_22.04 $

> apptainer version 1.3.4 (tried with additionally installing apptainer-suid)

>

> AP is HTCondor 23.x.

>

> Singularity related config:

> SINGULARITY_RUN_TEST_BEFORE_JOB = False

> # SINGULARITY_IS_SETUID = True

> # SINGULARITY_USE_PID_NAMESPACES = True

>

> SINGULARITY_BIND_EXPR = strcat(ifThenElse(AcctGroup == "chair_valera", "/share_chairvalera ", ""), \

>   ifThenElse(AcctGroup == "chair_ilg", "/share_chairilg ", ""), \

>   ifThenElse(WantScratchMounted isnt Undefined && WantScratchMounted, "/scratch ",""), \

>   ifThenElse(WantGPUHomeMounted isnt Undefined && WantGPUHomeMounted, "/home ", ""))

>

> SINGULARITY_VERBOSITY=-v

>

>

> More info w.r.t. to the points above.

> *condor_ssh_to_job or just interactive jobs:*

> See log:

> https://kingsx.cs.uni-saarland.de/index.php/s/yA5d7DqLsfjiSTK

>

> The job seems to successfully start and everything cgroup related seems to work out fine as well, however when a condor_ssh_to_job request comes in, things start to break down:

> > 05/21/25 14:30:21 ProcFamilyDirectCgroupV2::track_family_via_cgroup error writing to /sys/fs/cgroup/system.slice/htcondor/condor_raid_condor_lib_condor_execute_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxx/cgroup.subtree_control: Device or resource busy

> > 05/21/25 14:30:21 Error setting cgroup memory limit of 107374182400 in cgroup /sys/fs/cgroup/system.slice/htcondor/condor_raid_condor_lib_condor_execute_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxx/sshd: No such file or directory

> > 05/21/25 14:30:21 Error setting cgroup swap limit of 107374182400 in cgroup /sys/fs/cgroup/system.slice/htcondor/condor_raid_condor_lib_condor_execute_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxx/sshd: No such file or directory

> > 05/21/25 14:30:21 Error setting cgroup cpu weight of 1200 in cgroup /sys/fs/cgroup/system.slice/htcondor/condor_raid_condor_lib_condor_execute_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxx/sshd: No such file or directory

> > 05/21/25 14:30:21 Error enabling per-cgroup oom killing: 2 (No such file or directory)

> > 05/21/25 14:30:21 cgroup v2 could not attach gpu device limiter to cgroup: Operation not permitted

> Further down, the log is spammed with

> > 05/21/25 14:30:26 ProcFamilyDirectCgroupV2::get_usage cannot open /sys/fs/cgroup/system.slice/htcondor/condor_raid_condor_lib_condor_execute_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxxx/sshd/memory.stat: 2 No such file or directory

> > 05/21/25 14:30:26 error polling family usage

>

> Can you give any information on what is happening here/what I might have to fix in the set up?

>

> *Apptainer & GPUs*

> When requesting GPUs with the apptainer job, I can see that --nv is being added to the commandline, as expected.

> However, I don't see the required files being mapped (e.g. nvidia-smi) and thus I can't see the Nvidia GPUs.

> Looking at the stderr log with -v added to the singularity command line, I can see the following messages:

>

>

> > VERBOSE: persistenced socket /var/run/nvidia-persistenced/socket not found

> > WARNING: Could not find any nv files on this host!

>

> However, running the command from the log manually, works without any issue (while still producing the persistenced message):

> > /usr/bin/singularity -v exec -S /tmp -S /var/tmp -W /raid/condor/lib/condor/execute/dir_173820 --pwd /raid/condor/lib/condor/execute/dir_173820 -B /raid/condor/lib/condor/execute/dir_173820 --nv -B /scratch -B /home -B /etc/OpenCL/vendors --home /raid/condor/lib/condor/execute/dir_173820 -C /home/jmeyer/condor_tutorial/pytorch_24.02-py3.sif bash

>

> Any clues what is happening here?

>

> (Note: setting the apptainer.conf option "nvidia-container-cli", which reminds me of how docker handles stuff, doesn't seem to help either at the moment)

>

> *HOME environment variable:*

> The --home argument in the singularity command line seems to override my HOME environment variable as well. This is kinda annoying when mounting the shared file system (/home) where my actual home is - we even automatically add HOME to the getenv as a submit transformation to ensure users get their HOME variable set as they would expect it.

> Can we maybe override this behavior?

>

> Appreciate any help!

> Thanks,

> - Joachim Meyer

>

>
--

Joachim Meyer

HPC-Koordination & Support

UniversitÃt des Saarlandes

FR Informatik | HPC

Postanschrift: Postfach 15 11 50 | 66041 SaarbrÃcken

Besucheranschrift: Campus E1 3 | Raum 4.03

66123 SaarbrÃcken

T: +49 681 302-57522

jmeyer@xxxxxxxxxxxxxxxxxx

www.uni-saarland.de