Re: [HTCondor-users] systemd: drop-ins for Condor job slice?

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi Greg,

it seems to be something different (or I am not really understanding 
what systemd is actually doing ð ). I have played a bit more, but the 
delegation seems to be innocent.

---


To be sure, to be in the correct systemd scope (I am not sure anymore, 
if entering the namespaces with nsenter is actually having an effect, 
i.e., when nsenetering the scope(?) of the PID of my ip-ping-test unit, 
there seems to be no network block taking place...)


As cross-check and to be sure to be in the correct scope(?)/context(??), 
I added another drop-in with pings as ExecStartPre to be sure, that 
these are in the same context (ð) as the following condor PIDs [1]. 
Luckily, for these pre execs the IPAddressDenies are applied.

---

In the next step, I tried to put the BASE_CGROUP explicitly into the 
condor service or into another slice outside.

When forcing the startd cgroup into the systemd slice, i.e., 
BASE_CGROUP=condor.service, the block is applied and I cannot ping from 
a condor_ssh_to_job/interactive job any blocked IPs [3] â

But when using parallel slice, e.g., BASE_CGROUP=condor.slice ("slice" 
instead of "service" for good measures), systemd seems not to care and 
the drop-in rules are not inherited(?) down the line... [4] â

Switching Delegation* on/off does not have an impact as far as I can 
see... :-/

--

Right now, I am not sure, if nsenter is actually covering all scopes(?).
I.e., when I enter on the execution point as root the a job shell's 
namespaces, I can reach the IPs - while in the "same"/original shell 
from the job from where I cannot ping (being in the condor.service 
slice). So my understanding is currently, that nsenter is doing 
namespaces (fair enough) but that the network filtering handed down by 
systemd is not on network namespaces(??), or?

Probably the easiest(?) solution would be to force BASE_CGROUP onto 
condor.service - but there I am unsure if there might be unexpected 
effects when putting the jobs under the actual service hierarchy...??

Cheers,
  Thomas



[1]
> systemctl cat condor.service
...
# /etc/systemd/system/condor.service.d/01-network-filter.conf
[Service]
# IPAddressDeny=8.8.8.8 8.8.4.4 2001:4860:4860::8888 2001:4860:4860::8844
IPAddressDeny=8.8.8.8 2001:4860:4860::8888 2001:4860:4860::8844

# /etc/systemd/system/condor.service.d/99_delegatefoo.conf
#[Service]
#Delegate=false
# DelegateSubgroup=leaf

# /etc/systemd/system/condor.service.d/99_pingpre.conf
[Service]
ExecStartPre=-/usr/bin/ping -c5 8.8.8.8
ExecStartPre=-/usr/bin/ping -c5 8.8.4.4


[2]
> journalctl  -xeu condor
May 13 15:02:52 batch-preprod001.desy.de ping[3811070]: PING 8.8.8.8 
(8.8.8.8) 56(84) bytes of data.
May 13 15:02:52 batch-preprod001.desy.de ping[3811070]: --- 8.8.8.8 ping 
statistics ---
May 13 15:02:52 batch-preprod001.desy.de ping[3811070]: 5 packets 
transmitted, 0 received, 100% packet loss, time 4095ms
May 13 15:02:52 batch-preprod001.desy.de ping[3811103]: PING 8.8.4.4 
(8.8.4.4) 56(84) bytes of data.
May 13 15:02:52 batch-preprod001.desy.de ping[3811103]: 64 bytes from 
8.8.4.4: icmp_seq=1 ttl=118 time=13.8 ms
May 13 15:02:53 batch-preprod001.desy.de ping[3811103]: 64 bytes from 
8.8.4.4: icmp_seq=2 ttl=118 time=14.3 ms
May 13 15:02:54 batch-preprod001.desy.de ping[3811103]: 64 bytes from 
8.8.4.4: icmp_seq=3 ttl=118 time=14.0 ms
May 13 15:02:55 batch-preprod001.desy.de ping[3811103]: 64 bytes from 
8.8.4.4: icmp_seq=4 ttl=118 time=15.0 ms
May 13 15:02:56 batch-preprod001.desy.de ping[3811103]: 64 bytes from 
8.8.4.4: icmp_seq=5 ttl=118 time=13.6 ms
May 13 15:02:56 batch-preprod001.desy.de ping[3811103]: --- 8.8.4.4 ping 
statistics ---
May 13 15:02:56 batch-preprod001.desy.de ping[3811103]: 5 packets 
transmitted, 5 received, 0% packet loss, time 4005ms
May 13 15:02:56 batch-preprod001.desy.de ping[3811103]: rtt 
min/avg/max/mdev = 13.574/14.129/14.967/0.476 ms
May 13 15:02:56 batch-preprod001.desy.de systemd[1]: Started Condor 
Distributed High-Throughput-Computing.


[3]
[grid@batch-preprod001 ~]$ condor_config_val BASE_CGROUP
condor.service
[grid@batch-preprod001 ~]$ cat /proc/$$/cgroup
0::/system.slice/condor.service/condor_var_lib_condor_execute_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxx/sshd
[grid@batch-preprod001 ~]$ ping -c5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
^C
--- 8.8.8.8 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4086ms

[grid@batch-preprod001 ~]$ ping -c5 8.8.4.4
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.
64 bytes from 8.8.4.4: icmp_seq=1 ttl=118 time=14.1 ms
64 bytes from 8.8.4.4: icmp_seq=2 ttl=118 time=14.3 ms
64 bytes from 8.8.4.4: icmp_seq=3 ttl=118 time=14.2 ms
64 bytes from 8.8.4.4: icmp_seq=4 ttl=118 time=14.2 ms
64 bytes from 8.8.4.4: icmp_seq=5 ttl=118 time=12.6 ms

--- 8.8.4.4 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 12.584/13.878/14.341/0.651 ms


[4]
[grid@batch-preprod001 ~]$ condor_config_val BASE_CGROUP
condor.slice
[grid@batch-preprod001 ~]$ cat /proc/$$/cgroup
0::/system.slice/condor.slice/condor_var_lib_condor_execute_slot1_1@xxxxxxxxxxxxxxxxxxxxxxxx/sshd
[grid@batch-preprod001 ~]$ ping -c5 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=10.1 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=10.2 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=117 time=10.5 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=117 time=9.94 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=117 time=9.47 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 9.471/10.054/10.501/0.342 ms
[grid@batch-preprod001 ~]$ ping -c5 8.8.4.4
PING 8.8.4.4 (8.8.4.4) 56(84) bytes of data.
64 bytes from 8.8.4.4: icmp_seq=1 ttl=118 time=14.2 ms
64 bytes from 8.8.4.4: icmp_seq=2 ttl=118 time=13.8 ms
64 bytes from 8.8.4.4: icmp_seq=3 ttl=118 time=14.2 ms
64 bytes from 8.8.4.4: icmp_seq=4 ttl=118 time=14.2 ms
64 bytes from 8.8.4.4: icmp_seq=5 ttl=118 time=12.1 ms

--- 8.8.4.4 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 12.094/13.690/14.224/0.816 ms

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature