Re: [HTCondor-users] Possible misbehavior when using UNDEFINED OAuth Services

[Jaime Frey <jfrey@xxxxxxxxxxx>]

We donât plan to support empty values in the item data when submitting a batch of jobs.

Your grouping of unrelated jobs into a single submission is not recommended, and keeping them as separate submissions shouldnât affect how they auto-cluster (assuming the matchmaking expressions donât reference the ClusterId or the time when the job was submitted).

One hacky solution that wouldnât require a big change in your software is to create a dummy credential and use it when use_oauth_services would otherwise be empty.

 - Jaime

On Feb 17, 2026, at 5:45âAM, Kenyi Hurtado Anampa <khurtado@xxxxxx> wrote:

Hi Jaime,

Thank you. It looks like we have to find a different way, then.

Is there any other recommended pattern to express âunset/disabledâ per job for use_oauth_services when using schedd.submit(..., itemdata=...), (besides sorting jobs so that the no-token ones come first)?

If not...given that condor_submit and the Python bindings with itemdata behave somewhat differently here, do you expect this case to be supported in the future, or should we just plan around? E.g.: If there is a new feature/fix coming, we would rather wait for it than implementing the sorting workaround.

Best regards,

Kenyi

On Fri, Feb 13, 2026 at 2:34âPM Jaime Frey <jfrey@xxxxxxxxxxx> wrote:

I learned something new today.

Empty values in the item-data for a job submission are not supported. Trying to use them doesnât always generate an error and how the system treats them can vary based on how you submit (condor_submit vs python bindings, whether late materialization is used).

 - Jaime

On Feb 12, 2026, at 4:15âPM, Jaime Frey via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

Hmm. Thereâs a difference in behavior between condor_submit and the python bindings here. I see the same behavior that you do with the bindings. Iâll have to dig into this some more.

The only workaround I see right now is to sort your list of dictionaries of itemdata so that ones with no tokens occur first.

 - Jaime

On Feb 12, 2026, at 2:10âPM, Kenyi Hurtado Anampa <khurtado@xxxxxx> wrote:

Dear Jaime,

Thank you for the suggestion! 

I tried this by modifying the value of use_oauth_services per job (either "cms" or an empty string), with the following results:

Case 1:

job1 = {'use_oauth_services': ''}
job2 = {'use_oauth_services': 'cms'}

Both part of the same clusterId

Result:

This works well job1 effectively does not append any tokens, and job2 does.

Case2:

job1 = {'use_oauth_services': 'cms'}
job2 = {'use_oauth_services': ''}

Results:

Here, both job1 and job2 are sent with tokens, e,g.:

OAuthServicesNeeded = "cms"
OAuthServicesNeeded = "cms"

Here is some context and more details on how to reproduce this:

Our submission agents send groups of jobs (e.g.: batches pulled from the central production queue) to Condor using the Python bindings via the schedd.submit with the itemdata feature, with a structure similar to the example below. Previously, we encountered an issue (ticket-3351) when submitting jobs as a list of dictionaries that had slightly different keys. Since then, we ensured that all items use the same set of keys (so far, the differences have only involved custom keys, and setting them to UNDEFINED for jobs that do not require them has worked well.). Recently, however, the experiment needs to evaluate the use of CMS tokens more on a per-job basis (since jobs running older XRootD versions do not support tokens), and we are trying to maintain consistent keys across all jobs while implementing this change.

Do you have any additional suggestions?

Below is a minimal example to reproduce the behavior:

### Submit example

import classad2 as classad
import htcondor2 as htcondor

sub = htcondor.Submit("""
        universe = vanilla
        executable = test.sh
        output = out.$(Cluster)-$(Process)
        log = log.$(Cluster).log
        """)

schedd = htcondor.Schedd()

jobAds = []
job1 = {'use_oauth_services': 'cms'}
job2 = {'use_oauth_services': ''}

jobAds = [job1, job2]


print("HTCondor2 version = %s" % htcondor.version())
print("jobAds len = %s" % len(jobAds))
submitRes = schedd.submit(sub, itemdata=iter(jobAds))
clusterId = submitRes.cluster()
print("ClusterId = %s" % clusterId)
print("submitRes: {0}".format(submitRes))
print("submitRes numprocs: {0}".format(submitRes.num_procs()))
print("submitRes first_proc: {0}".format(submitRes.first_proc()))
print("dir(submitRes): {0}".format(dir(submitRes)))
print("ClusterId: {}".format(submitRes.cluster()))

Best regards,

Kenyi

On Thu, Feb 12, 2026 at 11:05âAM Jaime Frey via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

Hi Alan.

The appropriate way to clear a previous setting of use_oauth_services in the submit description is to set it to an empty value, like this:

use_oauth_services =

That is treated the same as not setting it at all for any following queue statements.

 - Jaime

> On Feb 11, 2026, at 8:55âPM, Alan Malta <alanmalta@xxxxxxxxx> wrote:
>
> Hi Cole,
>
> Thank you for looking into this and confirming the undesired behavior.
> I wonder if you have any advice or suggestions on how this issue could
> be avoided? Otherwise, do you think this is a change that could be
> made in HTCondor?
>
> Best regards,
> Alan.
>
> On Thu, Feb 5, 2026 at 9:20âAM Cole Bollig via HTCondor-users
> <htcondor-users@xxxxxxxxxxx> wrote:
>>
>> Hi Alan,
>>
>> There is some subtleness here in the fact that the Job Description Language (JDL) is not the ClassAd language despite looking very similar, and some commands supporting commands supporting ClassAd expressions/behavior. The JDL is essentially a higher-level language full of commands that condor_submit translates into ClassAds to send to the AP. This is why setting use_oauth_services = UNDEFINED is resulting in the string 'UNDEFINED' and not the ClassAd language UNDEFINED. I am not sure how or if one can set the value to the ClassAd value of UNDEFINED. I will need to converse with others on the team.
>>
>> -Cole Bollig
>> ________________________________
>> From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Alan Malta <alanmalta@xxxxxxxxx>
>> Sent: Wednesday, February 4, 2026 2:42 PM
>> To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
>> Cc: Alan Malta Rodrigues <alan.malta@xxxxxxx>
>> Subject: [HTCondor-users] Possible misbehavior when using UNDEFINED OAuth Services
>>
>> Dear HTCondor experts,
>>
>> we are finalizing token integration into CMS production workflows, and
>> due to some old application versions that need to be supported, we
>> need to instrument the WMS job submission component to either enable
>> or disable OAuth, via:
>> """
>> use_oauth_services = cms
>> """
>> or
>> """
>> use_oauth_services = UNDEFINED
>> """
>>
>> However, when setting this job classad to UNDEFINED, it looks like it
>> parses it as any other string, instead of the reserved UNDEFINED word.
>> Note that, if I set it to `cms`, it works well - as this is one of the
>> credentials stored in condor credd.
>>
>> One option to workaround this issue would be to not set
>> `use_oauth_services` at all, but then we hit an issue that has been
>> discussed in another thread, see:
>> https://www-auth.cs.wisc.edu/lists/htcondor-users/2025-November/msg00029.shtml
>>
>> Would you have any recommendation on what could be done? Is it an
>> issue that should be resolved in HTCondor?
>>
>> I am also providing details of my setup and how to reproduce it,
>> please see [1] for more information.
>>
>> Thank you in advance.
>> Best regards,
>> Alan.
>>
>> [1]
>> ### job submission
>> xxx@vocmsxxx:tokens $ condor_submit submit3.jdl
>> Submitting job(s)
>> Attempting to get tokens for UNDEFINED
>> htgettoken: Failure getting token from https://urldefense.com/v3/__https://xxx.cern.ch:xxx__;!!Mak6IKo!PTpEvNI2yWShGGcLuvJb6R0TtYwXvIiGcu6J8T9RtWYr2fY70HH12iOneVnPqVkA6Wb2M8vOKtPOCAajE1g$
>> Authentication needed for UNDEFINED
>> Attempting OIDC authentication with https://urldefense.com/v3/__https://xxx.cern.ch:xxx__;!!Mak6IKo!PTpEvNI2yWShGGcLuvJb6R0TtYwXvIiGcu6J8T9RtWYr2fY70HH12iOneVnPqVkA6Wb2M8vOKtPOCAajE1g$
>> htgettoken: Initiating authentication to https://urldefense.com/v3/__https://xxx.cern.ch:xxx__;!!Mak6IKo!PTpEvNI2yWShGGcLuvJb6R0TtYwXvIiGcu6J8T9RtWYr2fY70HH12iOneVnPqVkA6Wb2M8vOKtPOCAajE1g$
>> failed: HTTPError: Forbidden: permission denied
>> condor_vault_storer: htgettoken failed
>> Failed to process job credential requests (1):
>> 'process_job_credentials(): invoking '/usr/bin/condor_vault_storer'
>> failed: 0 (Success)
>> '; BAILING OUT.
>>
>> ### condor version
>> xxx@vocmsxxx:tokens $ condor_version
>> $CondorVersion: 24.0.6 2025-03-05 BuildID: 790851 PackageID: 24.0.6-1 $
>> $CondorPlatform: x86_64_AlmaLinux9 $
>>
>> ### submit file
>> xxx@vocmsxxx:tokens $ cat submit3.jdl
>> Universe = vanilla
>> executable = test.sh
>> should_transfer_files = YES
>> request_memory = 1 GB
>> request_cpus = 1
>> request_disk = 1 GB
>> # Where to submit - cms specific configuration
>> requirements = stringListMember(GLIDEIN_CMSSite,DESIRED_Sites)
>> +DESIRED_Sites = "T1_US_FNAL,T2_CH_CERN"
>> accounting_group = analysis
>> +AccountGroup = analysis
>> Output = test.$(Cluster)-$(Process).out
>> Error = test.$(Cluster)-$(Process).err
>> Log = test.$(Cluster).log
>> use_oauth_services = UNDEFINED
>> Queue 1
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>>
>> The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>>
>> The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
>
> The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/


_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe

The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe

The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/