Re: [HTCondor-users] Problem with scitokens-cpp v1.3.0 and HTCondor-CEs

[Antonio Delgado Peris <Antonio.Delgado.Peris@xxxxxxx>]

Hi again,

 

We found the problem. The hosts that were failing had the $SEC_SCITOKENS_CACHE owned by root, and empty. The hosts that were working had it owned by condor and containing a `scitokens` directory. Apparently, previous versions either didnât use the cache or ignore it if it was not there. Newest version seems to hang there waiting for the cache until there is the timeout.

 

Changing ownership to condor solves the problem. Stefano also confirmed it worked for him.

 

Now, why was that dir owned by root? I donât know. If I delete the dir, condor creates it on the fly with condor ownership. It must have been there already before the package update, but I donât know how it happened.

 

Probably the package could do a better job bypassing the cache if it is not usable, or at least reporting the problem more clearly.

 

Cheers,

    Antonio

 

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Antonio Delgado Peris via HTCondor-users
Sent: Wednesday, February 18, 2026 10:56 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Cc: Antonio Delgado Peris <Antonio.Delgado.Peris@xxxxxxx>
Subject: [HTCondor-users] Problem with scitokens-cpp v1.3.0 and HTCondor-CEs

 

Hello!

 

At CERN, we have observed that some our CEs were failing all scitoken validations after the package scitokens-cpp was updated to v1.3.0. (and, apparently, after a restart of the service). The symptom is the following entry in the CE's SchedLog:

 

02/17/26 16:22:09 DC_AUTHENTICATE: reason for authentication failure: AUTHENTICATE:1006:exceeded 1771341689 deadline during authentication|SCITOKENS:2:Failed to verify token and generate ACLs: Timeout when loading the OIDC metadata.|AUTHENTICATE:1004:Failed to authenticate using IDTOKENS|AUTHENTICATE:1004:Failed to authenticate using FS|FS:1004:Unable to lstat(/tmp/FS_XXXT6KLMa)

 

We see the timeout error message comes from: scitokens-cpp/src/scitokens_internal.h Line 838 in bd686d1

"Timeout when loading the OIDC metadata.");

 

It hasn't affected all CEs only a few of them. We don't know what other condition triggers the problem. However, by downgrading the package to scitokens-cpp-1.1.3, the problem has been solved in all cases.

 

Has somebody else seen something similar? Any idea about what we could do to further debug this issue?

 

PS: I have also reported this at https://github.com/scitokens/scitokens-cpp/issues/202

 

Cheers,
Antonio