Re: [HTCondor-users] thoughts on HTCondor python bindings submit improvements

[Luke Kreczko <L.Kreczko@xxxxxxxxxxxxx>]

Hi,

On 25/04/18 02:45, Brian Bockelman wrote:

Hi TJ,

I think it would be extremely useful to have a way to easily take a submit file on disk and to submit it from python. ÂThat should be a standalone interface -- not 100% sure we should try to wedge it into the Submit object. ÂPerhaps something like this:

Yes, an HTCondor python package instead just the pure bindings, could go a long way (e.g rootpy vs PyROOT).
If you add DAG handling on top you can tackle the biggest reasons people write their own packages/wrappers to interact with HTCondor (imagine the productivity boost!).

with schedd.transaction() as txn:

  htcondor.submit_file(fp)

Yeah, I've seen examples like this in the docs as well.
There is probably a good internal reason to do it this way, but what about something more straight-forward for users?

with schedd.transaction() as t:
ÂÂÂ t.submit(<file object or dict>)

same interface for submitting from a dictionary or file - everything else is internal.
If the implementation uses args & kwargs, you can even think of

with schedd.transaction() as t:
ÂÂÂ t.submit(
ÂÂÂ ÂÂÂ executable='/bin/cat',
ÂÂÂ ÂÂÂ args=['file1', 'file2'],
ÂÂÂ ÂÂÂ ...
ÂÂÂ )
Having such a simple wrapper allows you to extend features under the hood without the need to change the underlying HTCondor libraries.

Anyway, just my two smallest units of your local currency.

Cheers,
Luke

?

Anyhow, I've always struggled with ways to combine condor submit files and python approaches. ÂOne idea may be to do something like this:

with schedd.transaction() as txn:

 Âfor foo in iterdata:

   sub.queue(txn, foo)

where foo is a dictionary-like object that expands any missing macros in the submit object.

Brian

On Apr 24, 2018, at 4:42 PM, John M Knoeller <johnkn@xxxxxxxxxxx> wrote:

I have been thinking about how to make HTCondor python bindings submit as capable as condor_submit,

Â

The main thing missing right now is that the python bindings supports only the simplest possible queue statement.

I think that the bindings needs to support the full range of queue options, but in a pythonic way, so Iâm thinking this

Â

1. change the htcondor.Submit class so it can be constructed using the text of a valid submit file. like this

Â

sub = hcondor.Submit(âââ

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ executable = /bin/echo

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ queue 2 args in ( one, two, three )

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ âââ)

Â

2. add a method to the htcondor.Submit class that lets you get an iterator from the queue itemdata, like this

Â

for item in sub.itemdata() :

print item

Â

which (given the submit file above) would print:

{âargsâ : âoneâ}

{âargsâ : âtwoâ}

{âargsâ : âthreeâ}

Â

3. add a method to the htcondor.Submit class that would queue jobs to the schedd using an iterator for the queue itemdata. Âlike this

Â

with schedd.transaction() as txn :Â

sub.queue_with_iter(txn, 2, iter(itemdata))

Â

ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ which would submit 2 jobs for each item returned by the iterator.

The iterator could be sub.itemdata(), or any iterator that returned an equivalent type of item â either a dict, or a string.

Â

The reason this would need to be a new queue method on the submit object is that the current queue method already

allows a third argument that is a list that is used to return the submitted job classads. ÂÂThis would conflict with adding

an optional third argument to be the queue itemdata iterator.ÂÂ Having a new method also allows the return type to be

change to be more pythonic â return a tuple of <clusterid,classad> for instance.

Â

4. (optionally) change the current queue method on the htcondor.Submit object to honor any QUEUE statement supplied in the constructor of the Submit object. It would be a runtime error to pass more than just the txn argument to the queue method in this case. (since the count argument would conflict)

Â

Thoughts?

Is anyone currently using the third argument to the queue() method of the current htcondor.Submit class?

Would anyone be harmed if I just removed that argument and replaced it with an optional iterator?

Â

-tj

Â

Â

Â

Â

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message toÂhtcondor-users-request@xxxxxxxxxxxÂwith a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/