Re: [HTCondor-users] Rationale for choices in python-binding daemon queries

[John M Knoeller <johnkn@xxxxxxxxxxx>]

The choice comes down to tradeoffs between performance/memory usage and potential harm to other users.   

For the history query,  the other end of the connection is a process that has a relatively small memory footprint.  It will be reading from a potentially huge file and putting ads into the socket one at a time, so for that an iterator makes the most sense.  It doesn't matter much to the system as a while if the python program is reading from the iterator rather slowly and doing a lot of process of each ad as it arrives. 

For the Schedd query,  the other end of the connection is a forked copy of the entire Schedd, which can have a very large memory footprint.   At large Schedd can only fork a few times before the system runs out of memory,   A python program that read from the socket slowly would be keeping that forked copy of the Schedd live for a long time, which is bad for the system as a whole.  So the best model there is to not give control back to python until all of the ads have been read.  So the Schedd query returns a list rather than an iterator.

-tj

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Jeff Templon <templon@xxxxxxxxx>
Sent: Monday, June 3, 2024 5:13 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Rationale for choices in python-binding daemon queries

 

Hi

I spent an hour scratching my head why my history queries had “disappeared” but not my query about running jobs.  It took me a long time to realise the Schedd query was returning a list of ads, while the history query returned a HistoryIterator.  And for that, the Dutch would say “op is op” (close to the American “supplies may be limited”).

Why the difference between these two?  I’m curious.  I don’t need a solution, I just used the iterator to create a list, which I use in the rest of the program.  I’m hoping for insight why the choice is different.

JT


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/