Re: [HTCondor-users] token submission fails with Condor{CE}23/ set up

[Brian Lin <brian.lin@xxxxxxxx>]

Hey Thomas,

Most of the helpful debugging info for authZ is actually in the 
`D_SECURITY:2` debug level and you're right, the ScheDD will have the 
most useful info so you'll want to set something like the following on 
the CE:

ÂÂÂ SCHEDD_DEBUG = $(SCHEDD_DEBUG) D_CAT D_SECURITY:2

Then when you run the ping commands, try this:

ÂÂÂ _condor_TOOL_DEBUG="D_CAT D_SECURITY:2" condor_ce_ping -debug 
-verbose -name grid-htc-ce03.desy.de -pool grid-htc-ce03.desy.de:9619 WRITE

Both of those items should give you the info that you need to diagnose 
the issue. Let us know how that goes.

Thanks,
Brian

On 4/9/24 08:39, Thomas Hartmann wrote:
> Hi all,
>
> I am struggling to get token authz jobs working on our new 
> Condor23/CE6 on EL9.
> CondorCE traces & pings fail with not much of an error message like 
> [1] with Sci/WLCGtokens set coming from my test client [2]. Installed 
> packages are as of [3].
>
> The thing is, that ping/trace are successful with the same token(s) 
> towards our old EL7/Condor5 CEs [4]. Both, CondorCE23 and Condor5, 
> share the same mappings and configs so that there should be not much 
> of a difference with respect to the authz from the outside.
>
> Unfortunately, the logs gave not been very helpful so far. I have 
> increased the logging to full debug for all daemons - but the audit 
> logs are more or less empty [6]
>
> The only hint for my authz attempts are in the Scheds log complaining 
> that any authz method failed - but not explicitly a cause why for 
> sci/wlcgtokens [7].
>
> Maybe somebody has an idea, where my authz set up migh got stuck?
>
> Cheers and thanks,
> Â Thomas
>
> [1.el9]
> > condor_ce_ping -verbose -name grid-htc-ce03.desy.de -pool 
> grid-htc-ce03.desy.de:9619 WRITE
> WARNING: Missing daemon argument, defaulting to SCHEDD.
> WRITE failed!
> AUTHENTICATE:1003:Failed to authenticate with any method
> AUTHENTICATE:1004:Failed to authenticate using SCITOKENS
> AUTHENTICATE:1004:Failed to authenticate using IDTOKENS
> AUTHENTICATE:1004:Failed to authenticate using FS
>
>
>
> [2]
> > export BEARER_TOKEN_FILE=/tmp/token_$(id -u)
> > cat /tmp/token_$(id -u) | cut -d "." -f 2 | base64 -d 2>/dev/null 
> | jq
> {
> Â "wlcg.ver": "1.0",
> Â "sub": "1234-456-7890",
> Â "aud": "https://wlcg.cern.ch/jwt/v1/any";,
> Â "nbf": 1712666469,
> Â "scope": "openid compute.create offline_access compute.read 
> compute.cancel compute.modify",
> Â "iss": "https://wlcg.cloud.cnaf.infn.it/";,
> Â "exp": 1712670069,
> Â "iat": 1712666469,
> Â "jti": "234-567-8901",
> Â "client_id": "345-678-9012"
> }
>
>
> [3]
> condor-23.0.6-1.el9.x86_64
> condor-stash-plugin-6.12.1-1.x86_64
> htcondor-ce-23.0.6-1.el9.noarch
> htcondor-ce-bdii-23.0.6-1.el9.noarch
> htcondor-ce-client-23.0.6-1.el9.noarch
> htcondor-ce-condor-23.0.6-1.el9.noarch
> python3-condor-23.0.6-1.el9.x86_64
>
>
>
> [4.el7]
> > condor_ce_ping -verbose -name grid-htcondorce2.desy.de -pool 
> grid-htcondorce2.desy.de:9619 WRITE
> WARNING: Missing daemon argument, defaulting to SCHEDD.
> Destination:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ schedd grid-htcondorce2.desy.de
> Remote Version:ÂÂÂÂÂÂÂÂÂÂÂÂÂ $CondorVersion: 9.0.15 Jul 20 2022 
> BuildID: 597761 PackageID: 9.0.15-1 $
> Local Version: $CondorVersion: 9.0.17 May 27 2023 
> BuildID: 649540 PackageID: 9.0.17-3 $
> Session ID:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ grid-htcondorce2:337507:1712666509:35
> Instruction:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ WRITE
> Command:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ 60021
> Encryption:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ AES
> Integrity:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ AES
> Authenticated using:ÂÂÂÂÂÂÂÂ SCITOKENS
> All authentication methods:Â FS,TOKEN,SCITOKENS,GSI,SSL
> Remote Mapping:ÂÂÂÂÂÂÂÂÂÂÂÂÂ desyusr003@xxxxxxxxxxxxxxxxxx
> Authorized:ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ TRUE
>
>
> [5]
> [root@grid-htc-ce03 config.d]# condor_ce_config_val ALL_DEBUG 
> SCHEDD_AUDIT_LOG COLLECTOR_AUDIT_LOG
> D_FULLDEBUG
> /var/log/condor-ce/Audit.d/AuditLog
> /var/log/condor-ce/Audit.d/CollectorAuditPayloadLog
> [root@grid-htc-ce03 config.d]# ls -all 
> /var/log/condor-ce/Audit.d/AuditLog 
> /var/log/condor-ce/Audit.d/CollectorAuditPayloadLog
> -rw-r--r--. 1 condor condor 0 Apr 5 11:15 
> /var/log/condor-ce/Audit.d/AuditLog
> -rw-r--r--. 1 condor condor 2480 Apr 9 14:26 
> /var/log/condor-ce/Audit.d/CollectorAuditPayloadLog
>
> [6]
> [root@grid-htc-ce03 config.d]# tail -n5 
> /var/log/condor-ce/Audit.d/AuditLog 
> /var/log/condor-ce/Audit.d/CollectorAuditPayloadLog
> ==> /var/log/condor-ce/Audit.d/AuditLog <==
>
> ==> /var/log/condor-ce/Audit.d/CollectorAuditPayloadLog <==
> 04/09/24 14:02:25 (cid:0) Audit payload maximum job hours: 72
> 04/09/24 14:06:10 (cid:0) Audit payload maximum job hours: 72
> 04/09/24 14:17:14 (cid:0) Audit payload maximum job hours: 72
> 04/09/24 14:23:25 (cid:0) Audit payload maximum job hours: 72
> 04/09/24 14:26:24 (cid:0) Audit payload maximum job hours: 72
>
>
> [7]
> 04/09/24 14:41:41 DC_AUTHENTICATE: required authentication of 
> 131.169.223.90 failed: AUTHENTICATE:1003:Failed to authenticate with 
> any method|AUTHENTICATE:1004:Failed to authenticate using 
> SCITOKENS|AUTHENTICATE:1004:Failed to authenticate using IDTOK
> ENS|AUTHENTICATE:1004:Failed to authenticate using FS|FS:1004:Unable 
> to lstat(/tmp/FS_XXX4NnGn9)
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/