Re: [HTCondor-users] condor_adstash hard limit

[Jason Patton <jpatton@xxxxxxxxxxx>]

Hi Robin,

When using the Python bindings, history queries are considered remote queries. I *think* the limit is set by the value of HISTORY_HELPER_MAX_HISTORY (which has a default of 10,000) in the schedd's config, but I haven't tested this myself. See if changing that allows condor_adstash to query for more ads?

Jason Patton

On Wed, Aug 4, 2021 at 9:11 AM Hofsaess, Robin (ETP) <robin.hofsaess@xxxxxxx> wrote:

-------------------------------------------

$ condor_version
$CondorVersion: 9.1.2 Jul 29 2021 BuildID: 552044 PackageID: 9.1.2-1 $
$CondorPlatform: x86_64_CentOS7 $
-------------------------------------------

Hi everyone,

I have a question concerning condor_adstash:

We want to put our complete condor history into an elasticsearch DB with condor_adstash.

Therefore, we changed: https://github.com/htcondor/htcondor/blob/db44d1948122d10b98bf101acfbd67c3f109d8f1/src/condor_scripts/adstash/history.py#L68

history_iter = schedd.history(
                constraint="true",
                projection=[],
                match = -1,  # changed
                #match=min(10000, args.schedd_history_max_ads),
                since=since_str
            )

to get all entries, according to: https://htcondor.readthedocs.io/en/v8_8/apis/python-bindings/api/htcondor.html

But then, we experience a hard limit of 50000 entries per Schedd.

Does anyone know, where this limit comes from?

Can this be changed?

It seems that the HistoryIterator (https://github.com/htcondor/htcondor/blob/6809756bb368f8c77c4e1d17730addb0a0f9940d/src/python-bindings/schedd.cpp#L435) is the reason, as it only has 50000 entries. But I don't see any obvious limitation.

Thanks for your help!

Cheers,

Robin

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/