[HTCondor-users] token submission fails with Condor{CE}23/ set up

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi all,

I am struggling to get token authz jobs working on our new Condor23/CE6 
on EL9.
CondorCE traces & pings fail with not much of an error message like [1] 
with Sci/WLCGtokens set coming from my test client [2]. Installed 
packages are as of [3].

The thing is, that ping/trace are successful with the same token(s) 
towards our old EL7/Condor5 CEs [4]. Both, CondorCE23 and Condor5, share 
the same mappings and configs so that there should be not much of a 
difference with respect to the authz from the outside.

Unfortunately, the logs gave not been very helpful so far. I have 
increased the logging to full debug for all daemons - but the audit logs 
are more or less empty [6]

The only hint for my authz attempts are in the Scheds log complaining 
that any authz method failed - but not explicitly a cause why for 
sci/wlcgtokens [7].

Maybe somebody has an idea, where my authz set up migh got stuck?

Cheers and thanks,
  Thomas

[1.el9]
> condor_ce_ping -verbose -name grid-htc-ce03.desy.de -pool 
grid-htc-ce03.desy.de:9619 WRITE
WARNING: Missing daemon argument, defaulting to SCHEDD.
WRITE failed!
AUTHENTICATE:1003:Failed to authenticate with any method
AUTHENTICATE:1004:Failed to authenticate using SCITOKENS
AUTHENTICATE:1004:Failed to authenticate using IDTOKENS
AUTHENTICATE:1004:Failed to authenticate using FS



[2]
> export BEARER_TOKEN_FILE=/tmp/token_$(id -u)
>  cat  /tmp/token_$(id -u)  | cut -d "." -f 2 | base64 -d 2>/dev/null 
| jq
{
  "wlcg.ver": "1.0",
  "sub": "1234-456-7890",
  "aud": "https://wlcg.cern.ch/jwt/v1/any";,
  "nbf": 1712666469,
  "scope": "openid compute.create offline_access compute.read 
compute.cancel compute.modify",
  "iss": "https://wlcg.cloud.cnaf.infn.it/";,
  "exp": 1712670069,
  "iat": 1712666469,
  "jti": "234-567-8901",
  "client_id": "345-678-9012"
}


[3]
condor-23.0.6-1.el9.x86_64
condor-stash-plugin-6.12.1-1.x86_64
htcondor-ce-23.0.6-1.el9.noarch
htcondor-ce-bdii-23.0.6-1.el9.noarch
htcondor-ce-client-23.0.6-1.el9.noarch
htcondor-ce-condor-23.0.6-1.el9.noarch
python3-condor-23.0.6-1.el9.x86_64



[4.el7]
> condor_ce_ping -verbose -name grid-htcondorce2.desy.de -pool 
grid-htcondorce2.desy.de:9619 WRITE
WARNING: Missing daemon argument, defaulting to SCHEDD.
Destination:                 schedd grid-htcondorce2.desy.de
Remote Version:              $CondorVersion: 9.0.15 Jul 20 2022 BuildID: 
597761 PackageID: 9.0.15-1 $
Local  Version:              $CondorVersion: 9.0.17 May 27 2023 BuildID: 
649540 PackageID: 9.0.17-3 $
Session ID:                  grid-htcondorce2:337507:1712666509:35
Instruction:                 WRITE
Command:                     60021
Encryption:                  AES
Integrity:                   AES
Authenticated using:         SCITOKENS
All authentication methods:  FS,TOKEN,SCITOKENS,GSI,SSL
Remote Mapping:              desyusr003@xxxxxxxxxxxxxxxxxx
Authorized:                  TRUE


[5]
[root@grid-htc-ce03 config.d]# condor_ce_config_val ALL_DEBUG 
SCHEDD_AUDIT_LOG COLLECTOR_AUDIT_LOG
D_FULLDEBUG
/var/log/condor-ce/Audit.d/AuditLog
/var/log/condor-ce/Audit.d/CollectorAuditPayloadLog
[root@grid-htc-ce03 config.d]# ls -all 
/var/log/condor-ce/Audit.d/AuditLog 
/var/log/condor-ce/Audit.d/CollectorAuditPayloadLog
-rw-r--r--. 1 condor condor    0 Apr  5 11:15 
/var/log/condor-ce/Audit.d/AuditLog
-rw-r--r--. 1 condor condor 2480 Apr  9 14:26 
/var/log/condor-ce/Audit.d/CollectorAuditPayloadLog

[6]
[root@grid-htc-ce03 config.d]# tail -n5 
/var/log/condor-ce/Audit.d/AuditLog 
/var/log/condor-ce/Audit.d/CollectorAuditPayloadLog
==> /var/log/condor-ce/Audit.d/AuditLog <==

==> /var/log/condor-ce/Audit.d/CollectorAuditPayloadLog <==
04/09/24 14:02:25 (cid:0) Audit payload maximum job hours: 72
04/09/24 14:06:10 (cid:0) Audit payload maximum job hours: 72
04/09/24 14:17:14 (cid:0) Audit payload maximum job hours: 72
04/09/24 14:23:25 (cid:0) Audit payload maximum job hours: 72
04/09/24 14:26:24 (cid:0) Audit payload maximum job hours: 72


[7]
04/09/24 14:41:41 DC_AUTHENTICATE: required authentication of 
131.169.223.90 failed: AUTHENTICATE:1003:Failed to authenticate with any 
method|AUTHENTICATE:1004:Failed to authenticate using 
SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using IDTOK
ENS|AUTHENTICATE:1004:Failed to authenticate using FS|FS:1004:Unable to 
lstat(/tmp/FS_XXX4NnGn9)

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature