Re: [HTCondor-users] thoughts on HTCondor python bindings submit improvements

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

> On Apr 25, 2018, at 10:19 AM, John M Knoeller <johnkn@xxxxxxxxxxx> wrote:
> 
> We *could* multiply the supplied queue count with the one in the submit file, but that isn't possible for the item data,
> so I think override would be the most consistent, If it's on you to supply the iterator, then you either get it from the Submit instance
> or ignore the one in the Submit instance and supply a different one.
> 
> sub = htcondor.Submit("""
>    executable = /bin/echo
>    queue 3 args from filename
> """)
> 

This could work.  The submit object would internally know whether there is a queue statement and act accordingly.

> # submit using the itemdata from the queue line
> with schedd.transaction() as txn : sub.queue(txn, sub.qcount, sub.itemdata())
> 

I think this looks a bit like magic.  Why do we need to expose the items from the queue statement?

I think there's a strong argument to have python able to cleanly submit from an unaltered submit file.

I don't see a strong argument to expose the complexity of the "queue" statement given the user controls the loop anyway.

> # submit using my own itemdata
> itemdata = [{'args':'foo'}, {'args':'bar'}]
> with schedd.transaction() as txn : sub.queue(txn, 1, iter(itemdata))
> 

This is equivalent to what I wrote:

for foo in iterdata:
  sub.queue(txn, foo)

because, of course, an iterator is an object that you repeatedly make a method call on.  You're just inverting who is calling the function, not limiting what the function can do.

> # this would be an error because the queue statement specified a different queue count
> # and a partial override could lead to an invalid submit.
> with schedd.transaction() as txn: sub.queue(txn, 4)
> 
> -----Original Message-----
> From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Dimitri Maziuk
> Sent: Tuesday, April 24, 2018 5:40 PM
> To: htcondor-users@xxxxxxxxxxx
> Subject: Re: [HTCondor-users] thoughts on HTCondor python bindings submit improvements
> 
> On 04/24/2018 05:34 PM, John M Knoeller wrote:
>> If you prefer to specify the itemdata as a native python iterator, that would work in this scheme. Just leave the queue line  out of the submit file and provide the iterator as a python list
> 
> The obvious question then is what happens when I supply both the queue
> line and the iterable. Cartesian product?
> 
> -- 
> Dimitri Maziuk
> Programmer/sysadmin
> BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/