Re: [HTCondor-users] set environment in job submit file

[Greg Thain <gthain@xxxxxxxxxxx>]

Christoph, et. al.

By default, the starter sets the HOME environment variable of the job to 
the home directory of the user it will run the job as, as indicated in 
the home directory field of the password entry. You can turn this off 
by setting

STARTER_SETS_HOME_ENV = false

Once that is off, the job can request the home directory be set to the 
scratch dir by setting

environment = "HOME=$$(CondorScratchDir)"

However, that may not be sufficient for the needs of an arbitrary job. 
Remember that, in Unix, the home directory concept is not implemented by 
the kernel, but rather by a set of user space conventions. There are 
two different mechanisms to indicate to a process what the "home 
directory" is. The first is the HOME environment variable. The second 
is the home entry (the sixth field) in the /etc/passwd (or ldap, or ...) 
database. With the HTCondor settings above, we change the HOME 
environment variable, but not the /etc/passwd entry. This may cause 
surprises for users and their jobs.

Another approach (available on newer HTCondors) is to use the starter's 
MOUNT_UNDER_SCRATCH to bind-mount the existing home directory path onto 
the scratch directory, by setting something like

MOUNT_UNDER_SCRATCH = OsHomeDir

and leaving the STARTER_SETS_HOME_ENV at the default. This way, the 
job's HOME env matches the /etc/passwd entry, so there are fewer surprises.

I believe that for container jobs, the container runtime will overwrite 
HOME directory in many cases, so we need to pay attention to the job 
universe as well.


-greg