Re: [HTCondor-users] IDTOKENS and cli

[Michael Thomas <wart@xxxxxxxxxxx>]

Replying to myself:

I found the docs that say that the tokens are loaded in lexical order, 
so I created /etc/condor/tokens.d/99-root with the appropriate READ 
auth, which then allowed me to use the condor_status CLI to query the 
cluster pool.

With that finally working, I tried again to compare why this schedd was 
requiring tokens, while the others were fine using the shared 
SEC_PASSWORD_FILE to generate an in-memory token (as Brian described in 
a different thread).  I blew away the entire /etc/condor/tokens.d 
directory on the new schedd and then tried again.

This time it worked.  No on-disk tokens required.  I have no idea why 
this didn't work properly from the very beginning.

I have to feature requests that would help with debugging similar issues 
in the future:

1) condor_ping -debug should include the source of the auth data, eg the 
path to the password file or token.

2) The collector logs, when the D_SECURITY logging level is enabled, 
should include the authorizations listed in the token that was presented 
by the client.

Regards,

--Mike

On 4/29/22 15:00, Michael Thomas wrote:
> Hi Jaime,
>
> I spent more time on this today. The condor_ping tool that Todd pointed 
> me to is very helpful. While I am now able to get the schedd talking to 
> the collector and unprivileged users can submit jobs, I still can't get 
> the root user to query the pool from the schedd.
>
> schedd# # condor_ping -type collector -table all
>  ÂÂÂÂÂÂÂÂ Instruction Authentication Encryption Integrity Decision Identity
>  ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ ALLOWÂÂÂÂÂÂ IDTOKENSÂÂÂÂÂÂÂ AESÂÂÂÂÂÂ AESÂÂÂ ALLOW 
> condor@xxxxxxxxxxxxxxxxxxxxxxxx
>  ÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ READÂÂÂÂÂÂÂÂÂÂ FAILÂÂÂÂÂÂ FAILÂÂÂÂÂ FAILÂÂÂÂ FAIL FAIL 
> (use -verbose for more info)
>  ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ WRITEÂÂÂÂÂÂÂÂÂÂ FAILÂÂÂÂÂÂ FAILÂÂÂÂÂ FAILÂÂÂÂ FAIL FAIL 
> (use -verbose for more info)
>  ÂÂÂÂÂÂÂÂÂ NEGOTIATORÂÂÂÂÂÂÂÂÂÂ FAILÂÂÂÂÂÂ FAILÂÂÂÂÂ FAILÂÂÂÂ FAIL FAIL 
> (use -verbose for more info)
>  ÂÂÂÂÂÂ ADMINISTRATORÂÂÂÂÂÂÂÂÂÂ FAILÂÂÂÂÂÂ FAILÂÂÂÂÂ FAILÂÂÂÂ FAIL FAIL 
> (use -verbose for more info)
>  ÂÂÂÂÂÂÂÂÂÂÂÂÂÂ OWNERÂÂÂÂÂÂÂÂÂÂ FAILÂÂÂÂÂÂ FAILÂÂÂÂÂ FAILÂÂÂÂ FAIL FAIL 
> (use -verbose for more info)
>  ÂÂÂÂÂÂÂÂÂÂÂÂÂ CONFIGÂÂÂÂÂÂÂÂÂÂ FAILÂÂÂÂÂÂ FAILÂÂÂÂÂ FAILÂÂÂÂ FAIL FAIL 
> (use -verbose for more info)
>  ÂÂÂÂÂÂÂÂÂÂÂÂÂ DAEMONÂÂÂÂÂÂÂÂÂÂ FAILÂÂÂÂÂÂ FAILÂÂÂÂÂ FAILÂÂÂÂ FAIL FAIL 
> (use -verbose for more info)
>  ÂÂÂ ADVERTISE_STARTDÂÂÂÂÂÂÂÂÂÂ FAILÂÂÂÂÂÂ FAILÂÂÂÂÂ FAILÂÂÂÂ FAIL FAIL 
> (use -verbose for more info)
>  ÂÂÂ ADVERTISE_SCHEDDÂÂÂÂÂÂÂÂÂÂ FAILÂÂÂÂÂÂ FAILÂÂÂÂÂ FAILÂÂÂÂ FAIL FAIL 
> (use -verbose for more info)
>  ÂÂÂ ADVERTISE_MASTERÂÂÂÂÂÂ IDTOKENSÂÂÂÂÂÂÂ AESÂÂÂÂÂÂ AESÂÂÂ ALLOW 
> condor@xxxxxxxxxxxxxxxxxxxxxxxx
>
> The logs on the collector show the authentication succeeding, but being 
> denied access:
>
> [...]
> 04/29/22 14:16:05 HANDSHAKE: handshake() - i am the server
> 04/29/22 14:16:05 HANDSHAKE: client sent (methods == 2048)
> 04/29/22 14:16:05 HANDSHAKE: i picked (method == 2048)
> 04/29/22 14:16:05 HANDSHAKE: client received (method == 2048)
> 04/29/22 14:16:05 PW.
> 04/29/22 14:16:05 Will return to DC to continue authentication..
> 04/29/22 14:16:05 PASSWORD: entered authenticate_continue, state==100
> 04/29/22 14:16:05 PW: Server receiving 1.
> 04/29/22 14:16:05 Received: 0, 31(condor@xxxxxxxxxxxxxxxxxxxxxxxx), 256
> 04/29/22 14:16:05 PW: Server fetching password.
> 04/29/22 14:16:05 getTokenSigningKey(): for id=POOL, pool=1 v84mode=0 
> reading /etc/condor/condor_cred
> 04/29/22 14:16:05 PW: Server generating rb.
> 04/29/22 14:16:05 PW: Server sending.
> 04/29/22 14:16:05 In server_send: 0.
> 04/29/22 14:16:05 Calculating hkt 'condor@xxxxxxxxxxxxxxxxxxxxxxxx' 
> (31), 'condor_pool@xxxxxxxxxxxxxxxxxxxxxxxx' (36).
> 04/29/22 14:16:05 Server send 'condor@xxxxxxxxxxxxxxxxxxxxxxxx', 
> 'condor_pool@xxxxxxxxxxxxxxxxxxxxxxxx', 256 256 20
> 04/29/22 14:16:05 PASSWORD: leaving authenticate_continue, state==101, 
> return=2
> 04/29/22 14:16:05 AUTHENTICATE: auth would still block
> 04/29/22 14:16:05 Will return to DC to continue authentication..
> 04/29/22 14:16:05 PASSWORD: entered authenticate_continue, state==101
> 04/29/22 14:16:05 PW: Server receiving 2.
> 04/29/22 14:16:05 PW: Server checking hk.
> 04/29/22 14:16:05 In calculate_hk.
> 04/29/22 14:16:05 Setting session key.
> 04/29/22 14:16:05 Key length: 32
> 04/29/22 14:16:05 PW: Server set session key.
> 04/29/22 14:16:05 PASSWORD: leaving authenticate_continue, state==101, 
> return=1
> 04/29/22 14:16:05 Authentication was a Success.
> 04/29/22 14:16:05 AUTHENTICATION: setting default map to 
> condor@xxxxxxxxxxxxxxxxxxxxxxxx
> 04/29/22 14:16:05 AUTHENTICATION: post-map: current FQU is 
> 'condor@xxxxxxxxxxxxxxxxxxxxxxxx'
> 04/29/22 14:16:05 AUTHENTICATE: Exchanging keys with remote side.
> 04/29/22 14:16:05 AUTHENTICATE: Result of end of authenticate is 1.
> 04/29/22 14:16:05 DC_AUTHENTICATE: authentication of 10.13.5.58 complete.
> 04/29/22 14:16:05 DC_AUTHENTICATE: encryption enabled for session 
> ldas-condor:215706:1651259765:6878
> 04/29/22 14:16:05 DC_AUTHENTICATE: message authenticator enabled with 
> key id ldas-condor:215706:1651259765:6878.
> 04/29/22 14:16:05 DC_AUTHENTICATE: Success.
> 04/29/22 14:16:05 DC_AUTHENTICATE: authentication of <10.13.5.58:5037> 
> was successful but resulted in a limited authorization which did not 
> include this command (60020 DC_NOP_READ), so aborting.
> 04/29/22 14:16:05 DC_AUTHENTICATE: Command not authorized, done!
>
> ...even though the collector is supposed to allow read from everyone:
>
> # condor_config_val -dump ALLOW_
> # Configuration from machine: ldas-condor.ldas.ligo-la.caltech.edu
>
> # Parameters with names that match ALLOW_:
> ALLOW_ADMIN_COMMANDS = true
> ALLOW_ADMINISTRATOR = root@*, condor@*, condor_pool@*, schedd@$(UID_DOMAIN)
> ALLOW_CLIENT = *
> ALLOW_DAEMON = condor@*, condor_pool@*, schedd@$(UID_DOMAIN)
> ALLOW_NEGOTIATOR = condor@*, condor_pool@*, schedd@$(UID_DOMAIN)
> ALLOW_READ = *
> ALLOW_WRITE = *
> DAGMAN_ALLOW_ANY_NODE_NAME_CHARACTERS = false
> DAGMAN_ALLOW_EVENTS = 114
> FS_ALLOW_UNSAFE = false
> NEGOTIATOR_ALLOW_QUOTA_OVERSUBSCRIPTION = true
> SCHEDD_ALLOW_LATE_MATERIALIZE = true
> SEC_SCITOKENS_ALLOW_EXTRA_SLASH = false
> SUBMIT_ALLOW_GETENV = true
> XEN_ALLOW_HARDWARE_VT_SUSPEND = false
> [...]
>
>
> After some strace digging, I found that the cli is loading the token 
> /etc/condor/tokens.d/MASTER_auto_generated_token, which a jwt decoder 
> confirms has limited privileges:
>
> "scope": "condor:/ADVERTISE_MASTER",
>
> A token that I created with 'condor_token_create' on the schedd has more 
> privileges, but condor_ping seems to want to use these auth-generated 
> tokens instead.
>
> How can I allow the root user to default to using a token with more 
> privileges?
>
> --Mike
>
> On 4/27/22 11:14, Jaime Frey wrote:
> > To allow users to query the collector (i.e. use condor_status) without 
> > a token, we suggest allowing the ANONYMOUS or CLAIMTOBE authentication 
> > method for the READ and CLIENT authorization levels. If you donât set 
> > the authentication methods, CLAIMTOBE is enabled by default. If you 
> > use the get_htcondor tool to configure your machines, ANONYMOUS is 
> > enabled.
> >
> > If youâre setting the authentication methods explicitly on your config 
> > file, then try using these values:
> >
> > SEC_READ_AUTHENTICATION_METHODS = IDTOKENS, FS, ANONYMOUS
> > SEC_CLIENT_AUTHENTICATION_METHODS = IDTOKENS, FS, ANONYMOUS
> >
> > The -authz argument to condor_token_create specifies authorization 
> > levels, not command names. UPDATE_SCHEDD_AD, QUERY_STARTD_ADS, and 
> > UPDATE_AD_GENERIC are command names and thus wonât do anything in the 
> > token.
> > The authorization levels are described here: 
> > https://htcondor.readthedocs.io/en/latest/admin-manual/security.html#access-level-descriptions 
> >
> >
> > Â - Jaime
> >
> > On Apr 25, 2022, at 4:46 PM, Michael Thomas 
> > <wart@xxxxxxxxxxx<mailto:wart@xxxxxxxxxxx>> wrote:
> >
> > I'm still flailing about trying to get idtokens working on a new 
> > 9.0.12 schedd in my cluster.
> >
> > I was finally able to get the schedd talking to the collector by 
> > issuing a new token with condor_token_create (and a long list of 
> > permissions), then signing it on the CM:
> >
> > condor_token_create -identity 
> > schedd@xxxxxxxxxxxxxxxxxxxxxxxx<mailto:schedd@xxxxxxxxxxxxxxxxxxxxxxxx> -authz 
> > DAEMON -authz UPDATE_SCHEDD_AD -authz READ -authz WRITE -authz 
> > QUERY_STARTD_ADS -authz UPDATE_AD_GENERIC -authz ADMINISTRATOR
> >
> > However, I'm unable to get some CLI commands to work. Notably, the 
> > 'condor_status' command, when run as root, returns:
> >
> > # condor_status
> > Error: communication error
> > SECMAN:2010:Received "DENIED" from server for user 
> > condor@xxxxxxxxxxxxxxxxxxxxxxxx<mailto:condor@xxxxxxxxxxxxxxxxxxxxxxxx> using 
> > method IDTOKENS.
> >
> > The collector reports the following error:
> >
> > 04/25/22 16:41:03 DC_AUTHENTICATE: message authenticator enabled with 
> > key id ldas-condor:2734:1650922863:33339.
> > 04/25/22 16:41:03 DC_AUTHENTICATE: Success.
> > 04/25/22 16:41:03 DC_AUTHENTICATE: authentication of 
> > <10.13.5.58:20190> was successful but resulted in a limited 
> > authorization which did not include this command (5 QUERY_STARTD_ADS), 
> > so aborting.
> >
> > ...which seems odd considering that I explicitly included 
> > QUERY_STARTD_ADS in the token request above.
> >
> >
> > As a user, it won't even authenticate:
> >
> > $ condor_status
> > Error: communication error
> > AUTHENTICATE:1003:Failed to authenticate with any method
> > AUTHENTICATE:1004:Failed to authenticate using IDTOKENS
> > AUTHENTICATE:1004:Failed to authenticate using FS
> >
> > Any suggestions on where to look next?
> >
> > --Mike
> > _______________________________________________
> > HTCondor-users mailing list
> > To unsubscribe, send a message to 
> > htcondor-users-request@xxxxxxxxxxx<mailto:htcondor-users-request@xxxxxxxxxxx> 
> > with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/htcondor-users/
> >
> >
> > _______________________________________________
> > HTCondor-users mailing list
> > To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx 
> > with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> >
> > The archives can be found at:
> > https://lists.cs.wisc.edu/archive/htcondor-users/
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/