Re: [HTCondor-users] ssh/scp returning exit code 255 in Bash jobs

["Koschmieder, Lukas" <Lukas.Koschmieder@xxxxxxxxxxxxxxxxxxx>]

Hi,

In the meantime, I've discovered that this problem is actually caused by SELinux. All condor_starter processes are tagged as context=system_u:system_r:condor_startd_t:s0 and denied access /home/*/.ssh. Therefore, SSH-related commands that make use of this directory will fail. Everything will work as expected if you disable SELinux enforcing. I still need to find a way to include some kind of an exception for Condor jobs. Does anyone have an idea how to do that?

Cheers,

Lukas

--
Lukas Koschmieder
Steel Institute IEHK
RWTH Aachen University
Intzestraße 1
52072 Aachen
Germany

Tel: +49 (0)241 80 95823
Fax: +49 (0)241 80 92253
lukas.koschmieder@xxxxxxxxxxxxxxxxxxx

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Koschmieder, Lukas <Lukas.Koschmieder@xxxxxxxxxxxxxxxxxxx>
Sent: Friday, October 26, 2018 3:15:57 PM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] ssh/scp returning exit code 255 in Bash jobs

 

Hi,

- Do really want your job to deal with scripting to use scp and passing
around credentials? I think usually it is better/easier/safer to use
HTCondor's file transfer mechanism (which is scheduled) to move files
from your submit machine to the execute machine

I use Condor to spawn JupyterLab servers on remote machines. So my goal is to enable actual human users to run ssh, scp and condor_ssh_to_job on their JupyterLab servers. 

- If you need to use scp etc, the most common reason I've encountered
for an exit status of 255 in scripts that run outside of my home
environment is because I am contacting a host not in my
$HOME/.ssh/known_hosts file (almost certainly not the case if you are
using HTCondor without a shared file system or across administrative
domains).  To remedy, consider passing command-line switches to scp/ssh
like so:
   scp -o UserKnownHostsFile=./my_temporary_known_host ssh_host ...

$HOME/.ssh/know_hosts is definitely available.

Could this problem be somehow related to TTY?

Here are the results of a recent test:

Error file:

condor_exec.exe: cannot set terminal process group (-1): Inappropriate ioctl for device
condor_exec.exe: no job control in this shell
unknown option -- h
usage: ssh [-1246AaCfGgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]
           [-D [bind_address:]port] [-E log_file] [-e escape_char]
           [-F configfile] [-I pkcs11] [-i identity_file]
           [-J [user@]host[:port]] [-L address] [-l login_name] [-m mac_spec]
           [-O ctl_cmd] [-o option] [-p port] [-Q query_option] [-R address]
           [-S ctl_path] [-W host:port] [-w local_tun[:remote_tun]]
           [user@]hostname [command]

Job description file:

universe = vanilla
executable = /bin/bash
transfer_executable = false
arguments =" -li -c 'ssh -h' "
queue

Cheers,

Lukas

--
Lukas Koschmieder
Steel Institute IEHK
RWTH Aachen University
Intzestraße 1
52072 Aachen
Germany

Tel: +49 (0)241 80 95823
Fax: +49 (0)241 80 92253
lukas.koschmieder@xxxxxxxxxxxxxxxxxxx

---

From: Todd Tannenbaum <tannenba@xxxxxxxxxxx>
Sent: Tuesday, October 16, 2018 10:49:16 PM
To: HTCondor-Users Mail List; Koschmieder, Lukas
Subject: Re: [HTCondor-users] ssh/scp returning exit code 255 in Bash jobs

 

On 10/16/2018 9:51 AM, Koschmieder, Lukas wrote:
> Hi,
>
>
> When I try to run ssh/scp in a Condor Bash job, the command always fails
> with exit code 255. Are you familiar with this behavior? How do I enable
> ssh/scp in my Condor jobs?
>
>
> Cheers,
>
> Lukas
>

Couple quick thoughts:

- Do really want your job to deal with scripting to use scp and passing
around credentials? I think usually it is better/easier/safer to use
HTCondor's file transfer mechanism (which is scheduled) to move files
from your submit machine to the execute machine ... see
http://tinyurl.com/ya69km5j

- If you need to use scp etc, the most common reason I've encountered
for an exit status of 255 in scripts that run outside of my home
environment is because I am contacting a host not in my
$HOME/.ssh/known_hosts file (almost certainly not the case if you are
using HTCondor without a shared file system or across administrative
domains).  To remedy, consider passing command-line switches to scp/ssh
like so:
   scp -o UserKnownHostsFile=./my_temporary_known_host ssh_host ...

Hope the above helps
Todd