|
Hmm. Thereâs a difference in behavior between condor_submit and the python bindings here. I see the same behavior that you do with the bindings. Iâll have to dig into this some more.
The only workaround I see right now is to sort your list of dictionaries of itemdata so that ones with no tokens occur first.
- Jaime
On Feb 12, 2026, at 2:10âPM, Kenyi Hurtado Anampa <khurtado@xxxxxx> wrote:
Dear Jaime,
Thank you for the suggestion!
I tried this by modifying the value of use_oauth_services per job (either "cms" or an empty string), with the following results:
Case 1:
job1 = {'use_oauth_services': ''}
job2 = {'use_oauth_services': 'cms'}
Both part of the same clusterId
Result:
This works well job1 effectively does not append any tokens, and job2 does.
Case2:
job1 = {'use_oauth_services': 'cms'}
job2 = {'use_oauth_services': ''}
Results:
Here, both job1 and job2 are sent with tokens, e,g.:
OAuthServicesNeeded = "cms"
OAuthServicesNeeded = "cms"
Here is some context and more details on how to reproduce this:
Our submission agents send groups of jobs (e.g.: batches pulled from the central production queue) to Condor using the Python bindings via the schedd.submit with the itemdata feature, with a structure similar to the example below. Previously, we encountered
an issue (ticket-3351) when submitting jobs as a list of dictionaries that had slightly different keys. Since then, we ensured that all items use the same set of keys (so far, the differences have only involved custom keys, and setting them to
UNDEFINED for jobs that do not require them has worked well.). Recently, however, the experiment needs to evaluate the use of CMS tokens more on a per-job basis (since jobs running older XRootD versions do not support tokens), and we are trying
to maintain consistent keys across all jobs while implementing this change.
Do you have any additional suggestions?
Below is a minimal example to reproduce the behavior:
### Submit example
import classad2 as classad
import htcondor2 as htcondor
sub = htcondor.Submit("""
universe = vanilla
executable = test.sh
output = out.$(Cluster)-$(Process)
log = log.$(Cluster).log
""")
schedd = htcondor.Schedd()
jobAds = []
job1 = {'use_oauth_services': 'cms'}
job2 = {'use_oauth_services': ''}
jobAds = [job1, job2]
print("HTCondor2 version = %s" % htcondor.version())
print("jobAds len = %s" % len(jobAds))
submitRes = schedd.submit(sub, itemdata=iter(jobAds))
clusterId = submitRes.cluster()
print("ClusterId = %s" % clusterId)
print("submitRes: {0}".format(submitRes))
print("submitRes numprocs: {0}".format(submitRes.num_procs()))
print("submitRes first_proc: {0}".format(submitRes.first_proc()))
print("dir(submitRes): {0}".format(dir(submitRes)))
print("ClusterId: {}".format(submitRes.cluster()))
Best regards,
Kenyi
Hi Alan.
The appropriate way to clear a previous setting of use_oauth_services in the submit description is to set it to an empty value, like this:
use_oauth_services =
That is treated the same as not setting it at all for any following queue statements.
- Jaime
> On Feb 11, 2026, at 8:55âPM, Alan Malta <alanmalta@xxxxxxxxx> wrote:
>
> Hi Cole,
>
> Thank you for looking into this and confirming the undesired behavior.
> I wonder if you have any advice or suggestions on how this issue could
> be avoided? Otherwise, do you think this is a change that could be
> made in HTCondor?
>
> Best regards,
> Alan.
>
> On Thu, Feb 5, 2026 at 9:20âAM Cole Bollig via HTCondor-users
> <htcondor-users@xxxxxxxxxxx> wrote:
>>
>> Hi Alan,
>>
>> There is some subtleness here in the fact that the Job Description Language (JDL) is not the ClassAd language despite looking very similar, and some commands supporting commands supporting ClassAd expressions/behavior. The JDL is essentially a higher-level
language full of commands that condor_submit translates into ClassAds to send to the AP. This is why setting use_oauth_services = UNDEFINED is resulting in the string 'UNDEFINED' and not the ClassAd language UNDEFINED. I am not sure how or if one can set the
value to the ClassAd value of UNDEFINED. I will need to converse with others on the team.
>>
>> -Cole Bollig
>> ________________________________
>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Alan Malta <alanmalta@xxxxxxxxx>
>> Sent: Wednesday, February 4, 2026 2:42 PM
>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
>> Cc: Alan Malta Rodrigues <alan.malta@xxxxxxx>
>> Subject: [HTCondor-users] Possible misbehavior when using UNDEFINED OAuth Services
>>
>> Dear HTCondor experts,
>>
>> we are finalizing token integration into CMS production workflows, and
>> due to some old application versions that need to be supported, we
>> need to instrument the WMS job submission component to either enable
>> or disable OAuth, via:
>> """
>> use_oauth_services = cms
>> """
>> or
>> """
>> use_oauth_services = UNDEFINED
>> """
>>
>> However, when setting this job classad to UNDEFINED, it looks like it
>> parses it as any other string, instead of the reserved UNDEFINED word.
>> Note that, if I set it to `cms`, it works well - as this is one of the
>> credentials stored in condor credd.
>>
>> One option to workaround this issue would be to not set
>> `use_oauth_services` at all, but then we hit an issue that has been
>> discussed in another thread, see:
>>
https://www-auth.cs.wisc.edu/lists/htcondor-users/2025-November/msg00029.shtml
>>
>> Would you have any recommendation on what could be done? Is it an
>> issue that should be resolved in HTCondor?
>>
>> I am also providing details of my setup and how to reproduce it,
>> please see [1] for more information.
>>
>> Thank you in advance.
>> Best regards,
>> Alan.
>>
>> [1]
>> ### job submission
>> xxx@vocmsxxx:tokens $ condor_submit submit3.jdl
>> Submitting job(s)
>> Attempting to get tokens for UNDEFINED
>> htgettoken: Failure getting token from
https://urldefense.com/v3/__https://xxx.cern.ch:xxx__;!!Mak6IKo!PTpEvNI2yWShGGcLuvJb6R0TtYwXvIiGcu6J8T9RtWYr2fY70HH12iOneVnPqVkA6Wb2M8vOKtPOCAajE1g$
>> Authentication needed for UNDEFINED
>> Attempting OIDC authentication with
https://urldefense.com/v3/__https://xxx.cern.ch:xxx__;!!Mak6IKo!PTpEvNI2yWShGGcLuvJb6R0TtYwXvIiGcu6J8T9RtWYr2fY70HH12iOneVnPqVkA6Wb2M8vOKtPOCAajE1g$
>> htgettoken: Initiating authentication to
https://urldefense.com/v3/__https://xxx.cern.ch:xxx__;!!Mak6IKo!PTpEvNI2yWShGGcLuvJb6R0TtYwXvIiGcu6J8T9RtWYr2fY70HH12iOneVnPqVkA6Wb2M8vOKtPOCAajE1g$
>> failed: HTTPError: Forbidden: permission denied
>> condor_vault_storer: htgettoken failed
>> Failed to process job credential requests (1):
>> 'process_job_credentials(): invoking '/usr/bin/condor_vault_storer'
>> failed: 0 (Success)
>> '; BAILING OUT.
>>
>> ### condor version
>> xxx@vocmsxxx:tokens $ condor_version
>> $CondorVersion: 24.0.6 2025-03-05 BuildID: 790851 PackageID: 24.0.6-1 $
>> $CondorPlatform: x86_64_AlmaLinux9 $
>>
>> ### submit file
>> xxx@vocmsxxx:tokens $ cat submit3.jdl
>> Universe = vanilla
>> executable = test.sh
>> should_transfer_files = YES
>> request_memory = 1 GB
>> request_cpus = 1
>> request_disk = 1 GB
>> # Where to submit - cms specific configuration
>> requirements = stringListMember(GLIDEIN_CMSSite,DESIRED_Sites)
>> +DESIRED_Sites = "T1_US_FNAL,T2_CH_CERN"
>> accounting_group = analysis
>> +AccountGroup = analysis
>> Output = test.$(Cluster)-$(Process).out
>> Error = test.$(Cluster)-$(Process).err
>> Log = test.$(Cluster).log
>> use_oauth_services = UNDEFINED
>> Queue 1
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>>
>> The archives can be found at:
https://www-auth.cs.wisc.edu/lists/htcondor-users/
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>>
>> The archives can be found at:
https://www-auth.cs.wisc.edu/lists/htcondor-users/
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
>
> The archives can be found at:
https://www-auth.cs.wisc.edu/lists/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
The archives can be found at:
https://www-auth.cs.wisc.edu/lists/htcondor-users/
|
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}