Re: [HTCondor-users] custom universes?

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

Hi Klint,

It's not exactly what you asked for, but I'm working on something in the same ballpark:

https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=4804

The basic idea is the admin (or, more likely, packager) can write job transforms which are applied at submit time.  Then, the user would only need to put:

features = python26

in the config file and the "goop" to transform the job into bootstrapping and running the appropriate version of python is done inside the schedd.  This would allow very site-specific (UW has a "chtc_wrapper") or domain-specific (DMTCP-based checkpointing) functionality to appear like a first-class feature like universes.

The idea is still in the concept phase; an initial implementation exists, but there's not planned release.

Brian

> On Jan 7, 2015, at 6:51 PM, Klint Gore <kgore4@xxxxxxxxxx> wrote:
> 
> Just a thought - would there be a way to setup a python universe like the java universe?  Is there some threshold where an interpreter becomes a candidate for its own universe?  
> 
> Sooner or later, I'm going to have to deal with python across a mix of ubuntu and centos which have different default python versions.  If I could create a universes like python26, python27 and python3 that might make life easier.  I'm envisaging something like /etc/condor/universe.d/python27.conf that contains where the interpreter is, the environment setup for it particular to the node, and how to construct the line to execute the program.
> 
> Klint.
> 
> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Roger Herikstad
> Sent: Wednesday, 7 January 2015 12:49 PM
> To: HTCondor-Users Mail List
> Subject: Re: [HTCondor-users] Submit python on to windows host
> 
> 
> This is good enough for now. The reason I wanted to use the script as the executable initially is that I was hoping that the OS on the Windows machine would figure out how to run python script using its default python interpreter. In other words, I wanted a way to emulate the effect of adding the following 'shebang' on a unix machine #!/usr/bin/env python I realise that that's probably not a good idea anyway, since then I'd have no control over which packages are available on that particular machine. By specifying the path to a particular interpreter, I avoid that uncertainty. 
> Anyway, thanks again for the help.
> 
> ~ Roger
> On 6 Jan 2015, at 23:19, Ben Cotton wrote:
> 
>> Roger,
>> 
>> Offhand, I'm inclined to think the issue is that user condor-slot1 has 
>> a different environment than the user (you?) who runs testscript.py 
>> interactively. Specifically, I'm guessing that the Python interpreter 
>> is not in its path. If you set your executable to be 
>> C:\Python27\python.exe (or whatever is appropriate for your system) 
>> and use testscript.py as an argument, does that work?
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/