Re: [HTCondor-users] Upgrade from 8.8.5 to 9.0.17

[Vikrant Aggarwal <ervikrant06@xxxxxxxxx>]

Hello Todd,

Many thanks for your email.Â

To generate the TOKEN for schedd, I have set the auto-approve on negotiator node for token approval.Â

condor_token_request_auto_approve -netblock xx.xx.xx.0/24 -lifetime 3600

Aside note: ÂI don't see how we can query the conf done by above command later? may be somethingÂ"condor_token_requet_auto_approve list" can be helpful.Â

Removed POOL file from submitter which was earlier copied from CM. Did condor_reconfig to see whether schedd is sending a token request to CM or not, do they also require SSL setup between sched and CM (slide 11 in [1]) difference is I am talking about daemon-to-damon use case?

If I keep the POOL file on the submit node then it works like PASSWORD auth as you mentioned no IDTOKENS.Â

Even for using this command, we need to have POOL file on submitter.Â

# condor_token_create -identity condor@xxxxxxxxxxxxxxxx > /etc/condor/tokens.d/condor@xxxxxxxxxxxxxxxx

Following are the error logs from schedd when I don't have a POOL file.ÂÂ

01/05/23 03:57:19 (pid:3478132) SECMAN: required authentication with collector test.example.com failed, so aborting command UPDATE_SCHEDD_AD.
01/05/23 03:57:19 (pid:3478132) ERROR: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using SSL|AUTHENTICATE:1004:Failed to authenticate using GSI|GSI:5003:Failed to authenticate. Globus is reporting error (851968:560). There is probably a problem with your credentials. Â(Did you run grid-proxy-init?)|AUTHENTICATE:1004:Failed to authenticate using KERBEROS|AUTHENTICATE:1004:Failed to authenticate using FS
01/05/23 03:57:19 (pid:3478132) Collector update failed; will try to get a token request for trust domain test.example.com, identity (default).
01/05/23 03:57:19 (pid:3478132) Failed to start non-blocking update to <xx.xx.250.52:9618>.
01/05/23 03:57:20 (pid:3478132) AUTH_ERROR: Client not found in Kerberos database
01/05/23 03:57:20 (pid:3478132) authenticate_self_gss: acquiring self credentials failed. Please check your Condor configuration file if this is a server process. Or the user environment variable if this is a user process.

GSS Major Status: General failure
GSS Minor Status Error Chain:
globus_gsi_gssapi: Error with GSI credential
globus_gsi_gssapi: Error with gss credential handle

[1] https://agenda.hep.wisc.edu/event/1579/contributions/23053/attachments/7914/8949/HTCondor-Security-2021.pdf

Thanks & Regards,

Vikrant Aggarwal

On Tue, Jan 3, 2023 at 9:02 PM Todd L Miller via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:

> We are planning to upgrade the LTS htcondor version in our infra.

    Regarding your subject, the LTS version is now 10.0.0.

> I have gone through the documentation specifically about the steps of
> upgrading and HTCondor security. We are using Host Based security and
> flocking in our setup.

    I'm glad to hear that someone's been reading it! ;)

> - Default security setting is "use security:recommended_v9_0" which
> translates everything to user based instead of host based. Is
> recommended_v9_0 using TOKENID for authentication underneath?

    It is using IDTOKEN.

> - after starting the central manager, /etc/condor/passwords.d/POOL
> credential file is automatically generated, I had to copy this file on
> submit and worker nodes to make them communicate with the central manager
> node. This looks related to password based authentication. I don't see
> anything generated in /etc/condor/tokens.d/ for TOKENID based
> authentication? However I am able to create user token, is
> /etc/condor/passwords.d/POOL signing the JWT tokens? if yes, then in which
> scenario tokes.d directory will be used?

    For others following along at home: using `get_htcondor` will
avoid this issue entirely.

    The file `/etc/condor/passwords.d/POOL` is confusingly/misleading
named and implemented. HTCondor creates it so that every CM has a
"signing key", which is required to produce individual IDTOKENs.
Additionally, if a daemon has access to the same signing key as another
daemon, those two daemons will trust each other; the effect is the same
as the "pool password" method (hence the name).

    The idea for an IDTOKENS pool is that each machine in the pool is
issued an IDTOKEN by the CM. (There's some machinery available to make
the resulting distribution problem easier to solve if you trust your
network. Your daemon logs, prior to copying the signing key around,
should have mentioned that they requested an IDTOKEN, which you could
have approved at the CM.)Â This allows the daemons on that machine to
securely communicate with the CM. Note that users -- that is, the real
people submitting jobs -- can almost always get away with being
authenticated with FS at the submit node(s).

    Things are a little more annoying for the admin -- note that
daemons with IDTOKENS but not the signing key can't authenticate
_incoming_ connections. Most of the time, that won't matter; the
necessary authentication will have been securely forwarded through the
collector. But for some administrative commands, you'll either need the
signing key or a different authentication method.

    The `tokens.d` directory is used by HTCondor to store IDTOKENS for
the daemons.

> - Also as mentioned in [1] with flock based conf, password authentication
> will not work instead TOKENID, I didn't understand how it helps? Even with
> recommended_v9_0 I see only a POOL file is created. This will also become
> challenging for flock based conf, as I need to copy the same POOL file from
> two different pools on a single submit node?

    The idea for using IDTOKENS with flocking is that if schedd A is
flocking to CM B, then the administrator of CM B issues schedd A an
IDTOKEN and sends it (out of band) to the administrator of schedd A, who
copies into it into `tokens.d/`. (This allows A to securely flock to B
without involving any external authentication of either's identity.)

- ToddM
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/