Re: [HTCondor-users] Kerberos node authentication

[Andreas Haupt <andreas.haupt@xxxxxxx>]

Hi CMV,

On Wed, 2025-11-12 at 14:51 +0200, CMV wrote:
> On 11/12/25 10:55 AM, Andreas Haupt wrote:
> > > KERBEROS_MAP_FILE = /etc/condor/condor.kmap
> > What's the content of the mapping file?
> 
> cat /etc/condor/condor.kmap
> 
> EXAMPLE.COM = dev.example.com

OK, that looks correct.

> > > # KERBEROS_SERVER_PRINCIPAL = host/hts01.dev.example.com@xxxxxxxxxxx
> > > KERBEROS_CLIENT_PRINCIPAL = host/hts01.dev.example.com@xxxxxxxxxxx
> > That's most likely the default.
> 
> Thank you for the confirmation... it was a wild guess: the documentation 
> returns no results about that
> 
> https://htcondor.readthedocs.io/en/25.0/search.html#?q=KERBEROS_CLIENT_PRINCIPAL&check_keywords=yes&area=default

Then it's probably not used at all ;-)

The client principal name is build like this (to my knowledge):

$(KERBEROS_SERVER_SERVICE)/<fqdn.of.the.node>@<REALM>

... whereas <REALM> is a callout to the default_realm value in
/etc/krb5.conf

> > > 11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) KERBEROS: get remote
> > > server principal for "host/htm.dev.example.com"
> > > 11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) KERBEROS:
> > > krb5_unparse_name: host/htm.dev.example.com@
> > This looks weird: the realm part is missing. Does the domain->realm
> > mapping really work?
> > 
> > Could be a red herring, though.
> 
> That seemed like an interesting idea so I deleted the cache and then
> tried
> 
> sudo -u condor kinit -kt /etc/cndor/htm.dev.example.com.keytab 
> host/htm.dev.example.com
> 
> sudo -u condor klist
> 
> Ticket cache: KCM:995
> Default principal: host/htm.dev.example.com@xxxxxxxxxxx
> 
> Valid starting  ÂExpires      Service principal
> 11/12/25 13:25:35Â 11/12/25 23:25:35 krbtgt/EXAMPLE.COM@xxxxxxxxxxx
> ÂÂ Â Â Â renew until 11/19/25 13:25:35
> 
> 
> As you can see, even if I omit the realm from the kinit request the 
> library correctly picks the default configuration options from 
> /etc/krb5.conf.

Right, ok.

> Any other way to verify the domain->realm mapping?
> 
> > > 11/12/25 08:17:02 DC_AUTHENTICATE: required authentication of
> > > 192.168.10.61 failed: AUTHENTICATE:1003:Failed to authenticate with any
> > > method|AUTHENTICATE:1004:Failed to authenticate using KERBEROS
> > > 
> > > Any ideas would be greatly appreciated as I ran out of them.
> > I would also check KDC logs for wrong KRBTGT or TGT requests. The usual
> > things that break kerberos authentication are:
> > 
> > Â * clients build up principal names in the wrong way
> > Â * keys with wrong kvno in keytab
> 
> OK, here's what I see on the server when I delete the condor user's 
> cache and kinit again
> 
> Nov 12 14:31:30 keld-001.example.com krb5kdc[1997]: AS_REQ (4 etypes 
> {aes256-cts-hmac-sha1-96(18), aes128-cts-hmac-sha1-96(17), 
> aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19)}) 
> 192.168.10.50: ISSUE: authtime 1762950690, etypes 
> {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), 
> ses=aes256-cts-hmac-sha1-96(18)}, host/htm.dev.example.com@xxxxxxxxxxxÂ
> for krbtgt/EXAMPLE.COM@xxxxxxxxxxx
> 
> ... but! If I restart the condor daemon, nothing gets logged. Most 
> likely the condor daemon keeps the ticket in a file. Where would that be?

Could be, that tickets are only requested, when needed for
authentication. A developer can hopefully confirm.

htm is the manager node, I guess? I think, the authentication goes the
other way: AP is querying the CM. So in the logs you should see AS-REQ
attemts from the AP ip address for host/htm.dev.example.com@xxxxxxxxxxx

> Generally speaking, how can I increase condor's debug level to "Are you 
> nuts?" cos' right now I'm at the stage where I would like to see the 
> function call parameters.

Maybe?

ALL_DEBUG = D_FULLDEBUG

> > Kerberos works flawlessly here, settings are kept more or less to the
> > minimum:
> > 
> > ---
> > KERBEROS_MAP_FILE = /etc/condor/kerberos.map
> > KERBEROS_SERVER_KEYTAB = /etc/condor/krb5.keytab
> > KERBEROS_SERVER_SERVICE = condor
> > SEC_DEFAULT_AUTHENTICATION_METHODS = KERBEROS,IDTOKENS
> > ---
> > 
> > Maybe just take out most of the kerberos configuration from the
> > HTCondor config and keep system-wide stuff in /etc/krb5.conf only?
> 
> That's how I started: :-( With the very minimal and then kept adding 
> more stuff absurdly hoping it'll fix things.

:-( Good luck!


Cheers,
Andreas
-- 
| Andreas Haupt            | E-Mail: andreas.haupt@xxxxxxx
| DESY, Zeuthen            | WWW:    http://www.zeuthen.desy.de/~ahaupt
| Platanenallee 6          | Phone: +49/33762/7-7359
| D-15738 Zeuthen          |

Attachment: smime.p7s
Description: S/MIME cryptographic signature