On 3 Jun 2024, at 16:18, John M Knoeller via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:
The choice comes down to tradeoffs between performance/memory usage and potential harm to other users.
For the history query, the other end of the connection is a process that has a relatively small memory footprint. It will be reading from a potentially huge file and putting ads into the socket one at a time, so for that an iterator makes the most sense. It doesn't matter much to the system as a while if the python program is reading from the iterator rather slowly and doing a lot of process of each ad as it arrives.
For the Schedd query, the other end of the connection is a forked copy of the entire Schedd, which can have a very large memory footprint. At large Schedd can only fork a few times before the system runs out of memory, A python program that read from the socket slowly would be keeping that forked copy of the Schedd live for a long time, which is bad for the system as a whole. So the best model there is to not give control back to python until all of the ads have been read. So the Schedd query returns a list rather than an iterator.
-tj
From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Jeff Templon <templon@xxxxxxxxx>
Sent: Monday, June 3, 2024 5:13 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Rationale for choices in python-binding daemon queries
Hi
I spent an hour scratching my head why my history queries had âdisappearedâ but not my query about running jobs. It took me a long time to realise the Schedd query was returning a list of ads, while the history query returned a HistoryIterator. And for that, the Dutch would say âop is opâ (close to the American âsupplies may be limitedâ).
Why the difference between these two? Iâm curious. I donât need a solution, I just used the iterator to create a list, which I use in the rest of the program. Iâm hoping for insight why the choice is different.
JT
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-usersThe archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}