Re: [HTCondor-users] upgrading from 8.8.x to 9.0.4 - kerberos auth problems

[Oliver Freyermuth <freyermuth@xxxxxxxxxxxxxxxxxx>]

Hi,

we are using Kerberos with 8.8 right now, but seeing you found it's not an 9.0 related issue, maybe my input helps.

Seeing your log message:

> 08/18/21 08:13:43 init_daemon: Using default keytab FILE:/etc/krb5.keytab
> 08/18/21 08:13:43 init_daemon: Trying to get tgt credential for service host/server.dom.ain.edu@xxxxxxxxxxx <mailto:server.dom.ain.edu@xxxxxxxxxxx>
> 08/18/21 08:13:43 AUTH_ERROR: Client not found in Kerberos database

makes me think there is an issue using the host principal. Can you confirm that:
 kinit -k host/server.dom.ain.edu@xxxxxxxxxxx
works on the machine (and yields a TGT with "klist -Af")?

Also, you may want to check whether there are any SELinux denials related to accessing the keytab (or shortly disable it during the test).

Another issue we've hit starting from CentOS 8.4 is that, when using systemd-resolved (which is not the default yet), the forward and backward lookup of the local hostname do not match up (you don't get the full FQDN),
since they now use LLMNR by default. But that should have yielded a different error message.

For reference, our kerberos_mapfile contains (translated into your domains ;-) ):
 DOM.AIN.EDU = dom.ain.edu
and our certificate_maofile has:
 KERBEROS host/[^@]*@(.*) condor_pool@\1
 KERBEROS ([^/]*)/?[^@]*@(.*) \1@\2
We then use:
 UID_DOMAIN = dom.ain.edu
 ALLOW_DAEMON = condor@$(UID_DOMAIN), \
                condor@$(UID_DOMAIN)/*.$(UID_DOMAIN), \
                condor_pool@$(UID_DOMAIN), \
                condor_pool@$(UID_DOMAIN)/*.$(UID_DOMAIN), \
                $(FULL_HOSTNAME)

Cheers and hope this helps,
	Oliver

Am 18.08.21 um 17:29 schrieb Lee Damon:
> Hi Jaime,
>
> Here's all the changes I've made to the condor config between 8.8.13 and 9.0.4:
> add:
>  Â Â ALLOW_DAEMON Â Â Â Â Â Â Â Â Â Â= $(ALLOW_WRITE)
> remove:
>  Â HOSTALLOW_ADMINISTRATOR = $(CONDOR_HOST)
>  Â HOSTALLOW_READ = *.dom.ain.edu <http://dom.ain.edu>
>  Â HOSTALLOW_WRITE = *.dom.ain.edu <http://dom.ain.edu>
>  Â HOSTALLOW_NEGOTIATOR = $(CONDOR_HOST)
>  Â HOSTALLOW_NEGOTIATOR_SCHEDD = $(CONDOR_HOST)
>
> I've also tried adding a KERBEROS_MAP_FILE but that didn't seem to help (and I'm not even sure what to put in it. The other (working) 9.0.x install I have has a clear need to map a different dom.ain.) Plus, my currently working 8.8.x install doesn't use a map file.
>
> I first tried having the 9.0.4 client talk with the existing 8.8.13 pool but when these errors showed up I built a test collector host using 9.0.4. The errors are exactly the same regardless of which collector host is used.
>
> The error is showing up in MasterLog, SchedLog, and StartdLog. Here's what I find in SchedLog, the other two are exactly the same.
>
> I've redacted host & domain & IP but maintained case.
>
> 08/18/21 08:13:43 HANDSHAKE: in handshake(my_methods = 'KERBEROS')
> 08/18/21 08:13:43 HANDSHAKE: handshake() - i am the client
> 08/18/21 08:13:43 HANDSHAKE: sending (methods == 64) to server
> 08/18/21 08:13:43 HANDSHAKE: server replied (method = 64)
> 08/18/21 08:13:43 KERBEROS: get remote server principal for "host/server.dom.ain.edu <http://server.dom.ain.edu>"
> 08/18/21 08:13:43 KERBEROS: krb5_unparse_name: host/server.dom.ain.edu@xxxxxxxxxxx <mailto:server.dom.ain.edu@xxxxxxxxxxx>
> 08/18/21 08:13:43 KERBEROS: no user yet determined, will grab up to slash
> 08/18/21 08:13:43 KERBEROS: picked user: host
> 08/18/21 08:13:43 KERBEROS: remapping 'host' to 'condor'
> 08/18/21 08:13:43 Client is condor@xxxxxxxxxxx <mailto:condor@xxxxxxxxxxx>
> 08/18/21 08:13:43 init_daemon: client principal is 'host/client.dom.ain.edu@xxxxxxxxxxx <mailto:client.dom.ain.edu@xxxxxxxxxxx>'
> 08/18/21 08:13:43 init_daemon: Using default keytab FILE:/etc/krb5.keytab
> 08/18/21 08:13:43 init_daemon: Trying to get tgt credential for service host/server.dom.ain.edu@xxxxxxxxxxx <mailto:server.dom.ain.edu@xxxxxxxxxxx>
> 08/18/21 08:13:43 AUTH_ERROR: Client not found in Kerberos database
> 08/18/21 08:13:43 AUTHENTICATE: method 64 (KERBEROS) failed.
> 08/18/21 08:13:43 HANDSHAKE: in handshake(my_methods = '')
> 08/18/21 08:13:43 HANDSHAKE: handshake() - i am the client
> 08/18/21 08:13:43 HANDSHAKE: sending (methods == 0) to server
> 08/18/21 08:13:43 HANDSHAKE: server replied (method = 0)
> 08/18/21 08:13:43 SECMAN: required authentication with collector server.dom.ain.edu <http://server.dom.ain.edu> failed, so aborting command DC_START_TOKEN_REQUEST.
> 08/18/21 08:13:43 Failed to request a new token: DAEMON:1:failed to start command for token request with remote daemon at '<[IP-REDACTED]:9618?alias=server.dom.ain.edu <http://server.dom.ain.edu>>'.|AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using KERBEROS|AUTHENTICATE:1004:Failed to authenticate using FS
>
> nomad
>
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/


--
Oliver Freyermuth
UniversitÃt Bonn
Physikalisches Institut, Raum 1.047
NuÃallee 12
53115 Bonn
--
Tel.: +49 228 73 2367
Fax:  +49 228 73 7869
--

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature