Re: [HTCondor-users] thoughts on HTCondor python bindings submit improvements

[John M Knoeller <johnkn@xxxxxxxxxxx>]

If you prefer to specify the itemdata as a native python iterator, that would work in this scheme. Just leave the queue line  out of the submit file and provide the iterator as a python list

sent from my phone

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Sandeep Gupta <sandeep@xxxxxxxxxxxx>
Sent: Tuesday, April 24, 2018 5:05:33 PM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] thoughts on HTCondor python bindings submit improvements

 

Hi John,

I am not currently working with condor, but these features would be a welcome addition to the python bindings.

In the past, I raised this in a separate mail:

> Currently, is there anyway I can use this from python bindings as well? e.g.

>

> ```

> with schedd.transaction() as txn:

>   sub.queue(txn, njobs, numbers)

>```

>

> where numbers is a list I want to specify as an argument to each of the jobs. Of course, an invariant is `len(njobs) == len(numbers)`

```

sub = hcondor.Submit(“””

                executable = /bin/echo

                queue 2 args in ( one, two, three )

                “””)

```

I am not super sure if the above is the right way of doing things though. Because normally I would have (one, two, three) in some python variables, and doing string processing to get them into condor file format is not significantly different from constructing a temporary file, writing into it, and doing subprocess.check_output with `condor_submit`

Also not using the third argument to queue().

Thanks.

Sandeep.

On Tue, Apr 24, 2018 at 5:42 PM, John M Knoeller <johnkn@xxxxxxxxxxx> wrote:

I have been thinking about how to make HTCondor python bindings submit as capable as condor_submit,

 

The main thing missing right now is that the python bindings supports only the simplest possible queue statement.

I think that the bindings needs to support the full range of queue options, but in a pythonic way, so I’m thinking this

 

1. change the htcondor.Submit class so it can be constructed using the text of a valid submit file. like this

 

sub = hcondor.Submit(“””

                executable = /bin/echo

                queue 2 args in ( one, two, three )

                “””)

 

2. add a method to the htcondor.Submit class that lets you get an iterator from the queue itemdata, like this

 

for item in sub.itemdata() :

print item

 

which (given the submit file above) would print:

{‘args’ : ‘one’}

{‘args’ : ‘two’}

{‘args’ : ‘three’}

 

3. add a method to the htcondor.Submit class that would queue jobs to the schedd using an iterator for the queue itemdata.  like this

 

with schedd.transaction() as txn :

sub.queue_with_iter(txn, 2, iter(itemdata))

 

                which would submit 2 jobs for each item returned by the iterator.

The iterator could be sub.itemdata(), or any iterator that returned an equivalent type of item – either a dict, or a string.

 

The reason this would need to be a new queue method on the submit object is that the current queue method already

allows a third argument that is a list that is used to return the submitted job classads.   This would conflict with adding

an optional third argument to be the queue itemdata iterator.   Having a new method also allows the return type to be

change to be more pythonic – return a tuple of <clusterid,classad> for instance.

 

4. (optionally) change the current queue method on the htcondor.Submit object to honor any QUEUE statement supplied in the constructor of the Submit object.  It would be a runtime error to pass more than just the txn argument to the queue method in this case. (since the count argument would conflict)

 

Thoughts?

Is anyone currently using the third argument to the queue() method of the current htcondor.Submit class?

Would anyone be harmed if I just removed that argument and replaced it with an optional iterator?

 

-tj

 

 

 

 

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and are hereby notified that any disclosure, copying, or distribution of this message, or the taking of any action based on it, is strictly prohibited.