Re: [HTCondor-users] Conda env with workstation pool

[West Matthew <matthew.west@xxxxxxxx>]

Thanks all for the replies. I looked into conda-pack, which seems to be the most simplest solution for myself at the moment. I am looking into implementing some containers to make this sort of thing easier.

To make things a little more complex, each of these workstations does have a mounted shared network drive that acts as the place where our $HOME directories live. But when files are transferred to the local disk on  a random execute node in the pool, an

executable cannot see that $HOME due to lack of a Kerberos certificate.

I am worried that for folks not used to working with distributed systems, this need to tar/containerize/whatever their software before submission might scare them off. They are all used to homogeneous file systems where referencing their environment is "easy"

Cheers,
Matt

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Kandes, Martin <mkandes@xxxxxxxx>
Sent: Thursday, July 23, 2020 6:01:06 PM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Conda env with workstation pool

 

Hi Matt,

Yeah, I would likely go with Michael's proposed solution. Here is an example Singularity definition file that installs miniconda3 within the container [1].

Marty

[1]

https://github.com/mkandes/naked-singularity/blob/master/definition-files/comet/ubuntu/ubuntu.def

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Josh Karpel via HTCondor-users <htcondor-users@xxxxxxxxxxx>
Sent: Thursday, July 23, 2020 6:41 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Cc: Josh Karpel <karpel@xxxxxxxx>
Subject: Re: [HTCondor-users] Conda env with workstation pool

 

That pretty much lines up with what we tell people to do at CHTC: http://chtc.cs.wisc.edu/conda-installation.shtml

Josh Karpel

karpel@xxxxxxxx

On Thu, Jul 23, 2020 at 8:25 AM Michael Pelletier via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

I’ve built a Singularity definition file that installs Miniconda and creates an environment YAML file, then builds the environment, and then configures the container so that the environment is activated automatically at the startup of the Singularity container. With Miniconda a CUDA 10.2 and 18.04 Ubuntu Singularity container file is about 850 megabytes in size.

 

Singularity doesn’t necessarily have an extra infrastructure layer, as it doesn’t require any services from the host – I bet it would be possible to input-transfer the Singularity executable and run it on an input-transferred container.

 

Alternatively, you could build out the full virtualenv in a directory with Miniconda, and then input-transfer that whole directory and activate it when the job starts up, which would eliminate the need for modules to be available on the exec node.

 

Michael V Pelletier

Principal Engineer

Raytheon Technologies

Information Technology

Digital Transormation & Innovation

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of West Matthew
Sent: Thursday, July 23, 2020 7:03 AM
To: htcondor-users@xxxxxxxxxxx
Subject: [External] [HTCondor-users] Conda env with workstation pool

 

I am trying to run an analysis on my local workstation pool that relies on software in a conda virtual environment. When the job runs on a remote machine, it does not have access to the libraries in that env back on the submit machine.

Given that virtual environments are common practice when running locally, I an hoping there is some means of making python libraries accessible without resorting to an extra infrastructure layer like Docker.

 

Cheers,

Matt

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/