Re: [HTCondor-users] thoughts on HTCondor python bindings submit improvements

[John M Knoeller <johnkn@xxxxxxxxxxx>]

>> # submit using the itemdata from the queue line
>> with schedd.transaction() as txn : sub.queue(txn, sub.qcount, sub.itemdata())
>> 
> I think this looks a bit like magic.  Why do we need to expose the items from the queue statement?

So the script can interrogate / alter them of course.

> I think there's a strong argument to have python able to cleanly submit from an unaltered submit file.

yes.  I think that would be doing either this

   sub = htcondor.Submit(text_of_submit_file)
   with schedd.transaction as txn :  sub.queue(txn)

or something like this

   sub = htcondor.Submit(text_of_submit_file)
   with schedd.transaction as txn :  sub.queue_from_iter(txn, 1, sub.itemdata())

if we only do the first, then the submit file ends up being completely opaque to python, which is probably OK 90% of the time,
but an enormous pain when python needs to accept external submit files, but also needs to alter or understand them in limited ways.

You might want to ask a submit object if it will submit more than a single job, or submit on hold, or if all of the jobs share a set of input files. 

> I don't see a strong argument to expose the complexity of the "queue" statement given the user controls the loop anyway.

But having the user control the loop is a *problem*.     Late materialization moves that loop into the schedd. 

>> # submit using my own itemdata
>> itemdata = [{'args':'foo'}, {'args':'bar'}]
>> with schedd.transaction() as txn : sub.queue(txn, 1, iter(itemdata))
>> 
>This is equivalent to what I wrote:
>for foo in iterdata:
 > sub.queue(txn, foo)
>because, of course, an iterator is an object that you repeatedly make a method call on.  You're just inverting who is calling the function, not >limiting what the function can do.

This is only *logically* equivalent, the makes an enormous difference to the bindings code.  To replicate the behavior of condor submit, the bindings need to know when they are working on the last job in a cluster, so that either needs to be an argument to the queue statement, or the bindings need to be in control of the looping.   

-tj