[HTCondor-users] Kerberos node authentication

[CMV <ciprian.vizitiu@xxxxxxxxxxxxxxx>]

Hello everyone,

I'm trying to bring up a lab setup of a minimal htcondor using Alma 9.6 
VMs. Condor is installed from official RPM repos and I fully control 
both the DNS (which was made authoritative for the domains involved) and 
the (MIT) Kerberos server.

Initially I used IDTOKENS and it went OK I could see the manager 
mtm.dev.example.com accepting htx and hts nodes.I then deleted the 
tokens and changed the configuration(s) in order to make the manager and 
one of the submit nodes use Kerberos authentication instead but I 
haven't been able to make it work.

hts01.dev.example.com (the node) configuration


/etc/condor/config.d/01-submit.config

CONDOR_HOST = htm.dev.example.com
use role : get_htcondor_submit

SEC_DEFAULT_AUTHENTICATION_METHODS = KERBEROS
SEC_DEFAULT_AUTHENTICATION = REQUIRED

USE_KERBEROS = TRUE

KRB5_REALM = EXAMPLE.COM

KRB5_KDC = keld-001.example.com
KRB5_ADMIN_SERVER = keld-001.example.com

KERBEROS_MAP_FILE = /etc/condor/condor.kmap

# KERBEROS_SERVER_PRINCIPAL = host/hts01.dev.example.com@xxxxxxxxxxx
KERBEROS_CLIENT_PRINCIPAL = host/hts01.dev.example.com@xxxxxxxxxxx

# KERBEROS_SERVER_KEYTAB = /etc/condor/hts01.dev.example.com.keytab
KERBEROS_CLIENT_KEYTAB = /etc/condor/hts01.dev.example.com.keytab

# ALL_DEBUG = D_SECURITY, D_FULLDEBUG, D_COMMAND, D_NETWORK, D_GENERIC
ALL_DEBUG = D_ALL
SEC_DEBUG = D_SECURITY:4



The actual configuration at runtime on hts01.dev.example.com:


condor_config_val -dump 'K*' | grep -i Kerb

KERBEROS_CLIENT_KEYTAB = /etc/condor/hts01.dev.example.com.keytab
KERBEROS_CLIENT_PRINCIPAL = host/hts01.dev.example.com@xxxxxxxxxxx
KERBEROS_MAP_FILE = /etc/condor/condor.kmap
SEC_DEFAULT_AUTHENTICATION_METHODS = KERBEROS
USE_KERBEROS = TRUE

condor_config_val -dump 'K*' | grep SEC_

SEC_C_GAHP_WORKER_THREAD_DEFAULT_SESSION_DURATION = 1800
SEC_CLAIMTOBE_INCLUDE_DOMAIN = true
SEC_CLAIMTOBE_USER =
SEC_CLIENT_AUTHENTICATION = OPTIONAL
SEC_CLIENT_AUTHENTICATION_METHODS = 
$(SEC_DEFAULT_AUTHENTICATION_METHODS),ANONYMOUS
SEC_CREDENTIAL_REFRESH_INTERVAL = -1
SEC_CREDENTIAL_SWEEP_DELAY = 3600
SEC_CREDENTIAL_SWEEP_INTERVAL = 300
SEC_DEBUG = D_SECURITY:4
SEC_DEBUG_PRINT_KEYS = false
SEC_DEFAULT_AUTHENTICATION = REQUIRED
SEC_DEFAULT_AUTHENTICATION_METHODS = KERBEROS
SEC_DEFAULT_AUTHENTICATION_TIMEOUT = 20
SEC_DEFAULT_ENCRYPTION = required
SEC_DEFAULT_INTEGRITY = required
SEC_ENABLE_IMPERSONATION_TOKENS = false
SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = true
SEC_ENABLE_REMOTE_ADMINISTRATION = true
SEC_ENABLE_SCITOKEN_EXCHANGE = true
SEC_ENABLE_TOKEN_FETCH = true
SEC_ENABLE_TOKEN_REQUEST = true
SEC_IMPERSONATION_TOKEN_LIMITS =
SEC_INVALIDATE_SESSIONS_VIA_TCP = true
SEC_ISSUED_TOKEN_EXPIRATION =
SEC_PASSWORD_DIRECTORY = $(ETC)/passwords.d
SEC_PASSWORD_DOMAIN =
SEC_PASSWORD_FILE = $(SEC_PASSWORD_DIRECTORY)/POOL
SEC_READ_AUTHENTICATION = OPTIONAL
SEC_READ_AUTHENTICATION_METHODS = 
$(SEC_DEFAULT_AUTHENTICATION_METHODS),ANONYMOUS
SEC_SCITOKENS_ALLOW_FOREIGN_TOKEN_TYPES = true
SEC_SCITOKENS_CACHE = $(RUN)/cache
SEC_SCITOKENS_FOREIGN_TOKEN_ISSUERS = https://aai-dev.egi.eu/auth/realms/egi
SEC_SESSION_DURATION_SLOP = 20
SEC_SYSTEM_KNOWN_HOSTS = $(ETC)/known_hosts
SEC_TCP_SESSION_TIMEOUT = 20
SEC_TOKEN_AP_SIGNING_KEY_NAME = AP
SEC_TOKEN_DIRECTORY =
SEC_TOKEN_FETCH_ALLOWED_SIGNING_KEYS = POOL AP
SEC_TOKEN_ISSUER_KEY = POOL
SEC_TOKEN_MAX_AGE =
SEC_TOKEN_POOL_SIGNING_KEY_FILE = $(SEC_PASSWORD_FILE)
SEC_TOKEN_REQUEST_LIMITS =
SEC_TOKEN_REVOCATION_EXPR =
SEC_TOKEN_SYSTEM_DIRECTORY = $(ETC)/tokens.d
SEC_USE_FAMILY_SESSION = true


The keytab:

klist -k /etc/condor/hts01.dev.example.com.keytab
Keytab name: FILE:/etc/condor/hts01.dev.example.com.keytab
KVNO Principal
---- 
--------------------------------------------------------------------------
 Â2 host/hts01.dev.example.com@xxxxxxxxxxx
 Â2 host/hts01.dev.example.com@xxxxxxxxxxx
 Â2 host/hts01.dev.example.com@xxxxxxxxxxx

ls -la /etc/condor/hts01.dev.example.com.keytab
-rw------- 1 condor condor 239 Nov 11 12:53 
/etc/condor/hts01.dev.example.com.keytab


The clocks on the master, the submit and the Kerberos server are in sync 
but anyway I know that Kerberos infrastructure runs correctly because


sudo -u condor kinit -kt /etc/condor/hts01.dev.example.com.keytab

sudo -u condor klist
Ticket cache: KCM:995
Default principal: host/hts01.dev.example.com@xxxxxxxxxxx

Valid starting  ÂExpires      Service principal
11/11/25 15:53:59Â 11/12/25 01:53:59 krbtgt/EXAMPLE.COM@xxxxxxxxxxx
    renew until 11/18/25 15:53:59


Surprisingly enough, the debug report shows nothing about Kerberos


condor_status -debug

11/12/25 07:49:23 (fd:3) (pid:8852) (D_CONFIG) config: using subsystem 
'TOOL', local ''
11/12/25 07:49:23 (fd:3) (pid:8852) (D_CONFIG) Result of reading 
/etc/issue:Â \S
11/12/25 07:49:23 (fd:3) (pid:8852) (D_CONFIG) Result of reading 
/etc/redhat-release:Â AlmaLinux release 9.6 (Sage Margay)
11/12/25 07:49:23 (fd:3) (pid:8852) (D_LOAD) Reading from /proc/cpuinfo
11/12/25 07:49:23 (fd:3) (pid:8852) (D_LOAD) Found: Physical-IDs:True; 
Core-IDs:True
11/12/25 07:49:23 (fd:3) (pid:8852) (D_CONFIG) Using processor count: 2 
processors, 2 CPUs, 0 HTs
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) NETWORK_INTERFACE=* 
matches lo 127.0.0.1, enp1s0 192.168.10.61, choosing IP 192.168.10.61
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) hostname: 
hts01.dev.example.com
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) I am: hostname: hts01, 
fully qualified doman name: hts01.dev.example.com, IP: 192.168.10.61, 
IPv4: 192.168.10.61, IPv6:
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Trying to getting 
network interface information after reading config
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) NETWORK_INTERFACE=* 
matches lo 127.0.0.1, enp1s0 192.168.10.61, choosing IP 192.168.10.61
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) NETWORK_INTERFACE=* 
matches lo 127.0.0.1, enp1s0 192.168.10.61, choosing IP 192.168.10.61
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) hostname: 
hts01.dev.example.com
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) I am: hostname: hts01, 
fully qualified doman name: hts01.dev.example.com, IP: 192.168.10.61, 
IPv4: 192.168.10.61, IPv6:
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) COLLECTOR_HOST is set 
to "htm.dev.example.com"
11/12/25 07:49:23 (fd:3) (pid:8852) (D_DAEMONCORE) *** 
TIMEOUT_MULTIPLIER :: 0
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Checking if 
htm.dev.example.com is a sinful address
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) htm.dev.example.com is 
not a sinful address: does not begin with "<"
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) New Daemon obj 
(collector) name: "htm.dev.example.com", pool: "", addr: ""
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Using name 
"htm.dev.example.com" to find daemon
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Port not specified, 
using default (9618)
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Host info 
"htm.dev.example.com" is a hostname, finding IP address
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Found CM IP address and 
port <192.168.10.50:9618?alias=htm.dev.example.com>
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Daemon client 
(collector) address determined: name: "htm.dev.example.com", pool: 
"htm.dev.example.com", alias: "htm.dev.example.com", addr: 
"<192.168.10.50:9618?alias=htm.dev.example.com>"
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Querying collector 
<192.168.10.50:9618?alias=htm.dev.example.com> (htm.dev.example.com) 
with classad:
MyType = "Query"
Projection = "Activity Arch CondorLoadAvg EnteredCurrentActivity 
LastHeardFrom Machine Memory MyCurrentTime Name OpSys State"
Requirements = true
TargetType = "Machine"
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME)Â --- End of Query 
ClassAd ---
11/12/25 07:49:23 (fd:3) (pid:8852) (D_COMMAND) 
Daemon::startCommand(QUERY_STARTD_ADS,...) making connection to 
<192.168.10.50:9618?alias=htm.dev.example.com>
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Guess address string 
for host = <192.168.10.50:9618?alias=htm.dev.example.com>, port = 0
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) it was sinful string. 
ip = 192.168.10.50, port = 9618
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) CONNECT bound to 
<192.168.10.61:13763?alias=hts01.dev.example.com> fd=3 
peer=<192.168.10.50:9618>
11/12/25 07:49:23 (fd:3) (pid:8852) (D_SECURITY) SECMAN: command 5 
QUERY_STARTD_ADS to collector htm.dev.example.com from TCP port 13763 
(blocking).
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) Send Header contents: 01 
00 00 02 a9
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) AESGCM: Send digest 
added 5 + 681 bytes
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) condor_write(fd=3 
collector htm.dev.example.com,,size=686,timeout=60,flags=0,non_blocking=0)
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) condor_read(fd=3 
collector htm.dev.example.com,,size=5,timeout=60,flags=0,non_blocking=0)
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) condor_read(fd=3 
collector htm.dev.example.com,,size=522,timeout=60,flags=0,non_blocking=0)
11/12/25 07:49:23 (fd:3) (pid:8852) (D_SECURITY) SECMAN: generating AES 
key for session with collector htm.dev.example.com...
11/12/25 07:49:23 (fd:3) (pid:8852) (D_SECURITY) SECMAN: successfully 
enabled encryption!
11/12/25 07:49:23 (fd:3) (pid:8852) (D_SECURITY) SECMAN: successfully 
enabled message authenticator!
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) condor_read(fd=3 
collector htm.dev.example.com,,size=5,timeout=60,flags=0,non_blocking=0)
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) condor_read(fd=3 
collector htm.dev.example.com,,size=262,timeout=60,flags=0,non_blocking=0)
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) Expecting AAD with 
handshake digest b9 bd fb 4e 5c df ab 42 51 dc fb 53 f9 c7 51 6a ce 55 
33 23 a2 51 d8 14 f2 1d a4 29 73 29 2c 0f 85 ba fd 8a 12 02 8d ff 70 35 
c1 36 48 fe 38 f2 71 6d dc 40 ca 4a c6 9d e7 90 9f b7 e5 be e5 49 01 00 
00 01 06
11/12/25 07:49:23 (fd:3) (pid:8852) (D_SECURITY) SESSION: client 
duplicated AES to BLOWFISH key for UDP.
11/12/25 07:49:23 (fd:3) (pid:8852) (D_SECURITY) SECMAN: added session 
htm:11208:1762926567:7663 to cache for 60 seconds (3600s lease).
11/12/25 07:49:23 (fd:3) (pid:8852) (D_SECURITY) SECMAN: startCommand 
succeeded.
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) Sock::ciphertext_size: 
went from plaintext_size 244 to ciphertext_size 276.
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) Sending AAD with 
handshake digest 85 ba fd 8a 12 02 8d ff 70 35 c1 36 48 fe 38 f2 71 6d 
dc 40 ca 4a c6 9d e7 90 9f b7 e5 be e5 49 b9 bd fb 4e 5c df ab 42 51 dc 
fb 53 f9 c7 51 6a ce 55 33 23 a2 51 d8 14 f2 1d a4 29 73 29 2c 0f 01 00 
00 01 14
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) Resetting Header for send.
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) condor_write(fd=3 
collector htm.dev.example.com,,size=281,timeout=60,flags=0,non_blocking=0)
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) condor_read(fd=3 
collector htm.dev.example.com,,size=5,timeout=60,flags=0,non_blocking=0)
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) condor_read(fd=3 
collector htm.dev.example.com,,size=24,timeout=60,flags=0,non_blocking=0)
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) Resetting Header for recv.
11/12/25 07:49:23 (fd:3) (pid:8852) (D_NETWORK) CLOSE TCP 
<192.168.10.61:13763> fd=3
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Destroying Daemon object:
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) Type: 5 (collector), 
Name: htm.dev.example.com, Addr: 
<192.168.10.50:9618?alias=htm.dev.example.com>
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) FullHost: 
htm.dev.example.com, Host: htm, Pool: htm.dev.example.com, Port: 9618
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME) IsLocal: N, IdStr: 
collector htm.dev.example.com, Error:
11/12/25 07:49:23 (fd:3) (pid:8852) (D_HOSTNAME)Â --- End of Daemon 
object info ---


... the only relevant entries could be found in MasterLog and SchedLog


11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) CLOSE TCP 
<192.168.10.61:3081> fd=10
11/12/25 08:21:58 (fd:10) (pid:9048) (D_COMMAND) Return from Handler 
<SecManStartCommand::WaitForSocketCallback UPDATE_MASTER_AD> 0.009762s
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) Trying token request 
to remote host htm.dev.example.com for user (default).
11/12/25 08:21:58 (fd:10) (pid:9048) (D_COMMAND) 
Daemon::startTokenRequest() making connection to 
'<192.168.10.50:9618?alias=htm.dev.example.com>'
11/12/25 08:21:58 (fd:10) (pid:9048) (D_HOSTNAME) Guess address string 
for host = <192.168.10.50:9618?alias=htm.dev.example.com>, port = 0
11/12/25 08:21:58 (fd:10) (pid:9048) (D_HOSTNAME) it was sinful string. 
ip = 192.168.10.50, port = 9618
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) CONNECT bound to 
<192.168.10.61:22307?alias=hts01.dev.example.com> fd=10 
peer=<192.168.10.50:9618>
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) SECMAN: command 60047 
DC_START_TOKEN_REQUEST to collector htm.dev.example.com from TCP port 
22307 (blocking).
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) Send Header contents: 
01 00 00 03 29
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) AESGCM: Send digest 
added 5 + 809 bytes
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_write(fd=10 
collector htm.dev.example.com,,size=814,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_write(fd=10 
collector htm.dev.example.com,,size=814,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_read(fd=10 
collector htm.dev.example.com,,size=5,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_read(fd=10 
collector htm.dev.example.com,,size=516,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) SECMAN: new session, 
doing initial authentication.
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) SECMAN: Auth methods: 
KERBEROS
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) AUTHENTICATE: setting 
timeout for <192.168.10.50:9618?alias=htm.dev.example.com> to 20.
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) HANDSHAKE: in 
handshake(my_methods = 'KERBEROS')
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) HANDSHAKE: handshake() 
- i am the client
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) HANDSHAKE: sending 
(methods == 64) to server
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) Send Header contents: 
01 00 00 00 08
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) AESGCM: Send digest 
added 5 + 8 bytes
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_write(fd=10 
collector htm.dev.example.com,,size=13,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_read(fd=10 
collector htm.dev.example.com,,size=5,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_read(fd=10 
collector htm.dev.example.com,,size=8,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) HANDSHAKE: server 
replied (method = 64)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) KERBEROS: get remote 
server principal for "host/htm.dev.example.com"
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) KERBEROS: 
krb5_unparse_name: host/htm.dev.example.com@
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) KERBEROS: no user yet 
determined, will grab up to slash
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) KERBEROS: picked user: 
host
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) KERBEROS: remapping 
'host' to 'condor'
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) Send Header contents: 
01 00 00 00 08
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) AESGCM: Send digest 
added 5 + 8 bytes
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_write(fd=10 
collector htm.dev.example.com,,size=13,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) AUTHENTICATE: method 
64 (KERBEROS) failed.
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) HANDSHAKE: in 
handshake(my_methods = '')
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) HANDSHAKE: handshake() 
- i am the client
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) HANDSHAKE: sending 
(methods == 0) to server
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) Send Header contents: 
01 00 00 00 08
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) AESGCM: Send digest 
added 5 + 8 bytes
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_write(fd=10 
collector htm.dev.example.com,,size=13,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_read(fd=10 
collector htm.dev.example.com,,size=5,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) condor_read(fd=10 
collector htm.dev.example.com,,size=8,timeout=20,flags=0,non_blocking=0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_SECURITY) HANDSHAKE: server 
replied (method = 0)
11/12/25 08:21:58 (fd:10) (pid:9048) (D_ALWAYS) SECMAN: required 
authentication with collector htm.dev.example.com failed, so aborting 
command DC_START_TOKEN_REQUE
ST.
11/12/25 08:21:58 (fd:10) (pid:9048) (D_NETWORK) CLOSE TCP 
<192.168.10.61:22307> fd=10
11/12/25 08:21:58 (fd:10) (pid:9048) (D_ALWAYS) Failed to request a new 
token: DAEMON:1:failed to start command for token request with remote 
daemon at '<192.
168.10.50:9618?alias=htm.dev.example.com>'.|AUTHENTICATE:1003:Failed to 
authenticate with any method|AUTHENTICATE:1004:Failed to authenticate 
using KERBEROS
11/12/25 08:21:58 (fd:10) (pid:9048) (D_DAEMONCORE) In cancel_timer(), id=25
11/12/25 08:21:58 (fd:10) (pid:9048) (D_HOSTNAME) Destroying Daemon object:
11/12/25 08:21:58 (fd:10) (pid:9048) (D_HOSTNAME) Type: 5 (collector), 
Name: htm.dev.example.com, Addr: 
<192.168.10.50:9618?alias=htm.dev.example.com>
11/12/25 08:21:58 (fd:10) (pid:9048) (D_HOSTNAME) FullHost: 
htm.dev.example.com, Host: htm, Pool: htm.dev.example.com, Port: 9618
11/12/25 08:21:58 (fd:10) (pid:9048) (D_HOSTNAME) IsLocal: N, IdStr: 
collector htm.dev.example.com, Error:
11/12/25 08:21:58 (fd:10) (pid:9048) (D_HOSTNAME)Â --- End of Daemon 
object info ---


Of course I also tried with principals and keytabs containing "condor" 
instead of "host" (with the appropiate configuration changes) in fact 
this is how I started; same error. Out of sheer desperation I tried 
using the KERBEROS_SERVER_x config options instead of the CLIENT ones; 
still nothing. Tried

export KRB5_TRACE=/tmp/krb5.trace


... but neither a condor_status -debug nor a daemon started manually 
produced anything in /tmp/krb5.trace


On both the master and the submit node I have

cat /etc/condor/condor.kmap
EXAMPLE.COM = dev.example.com


... and the master configuration is:

CONDOR_HOST = htm.dev.example.com
use role : get_htcondor_central_manager

SEC_DEFAULT_AUTHENTICATION_METHODS = KERBEROS
SEC_DEFAULT_AUTHENTICATION = REQUIRED

USE_KERBEROS = TRUE

KRB5_REALM = EXAMPLE.COM

KRB5_KDC = keld-001.example.com
KRB5_ADMIN_SERVER = keld-001.example.com

KERBEROS_SERVER_KEYTAB = /etc/condor/htm.dev.example.com.keytab

KERBEROS_MAP_FILE = /etc/condor/condor.kmap

KERBEROS_SERVER_PRINCIPAL = host/htm.dev.example.com@xxxxxxxxxxx


... yet all I get in master logs is:

11/12/25 08:16:57 DC_AUTHENTICATE: required authentication of 
192.168.10.61 failed: AUTHENTICATE:1003:Failed to authenticate with any 
method|AUTHENTICATE:1004:Failed to authenticate using KERBEROS
11/12/25 08:16:57 DC_AUTHENTICATE: required authentication of 
192.168.10.61 failed: AUTHENTICATE:1003:Failed to authenticate with any 
method|AUTHENTICATE:1004:Failed to authenticate using KERBEROS
11/12/25 08:17:02 DC_AUTHENTICATE: required authentication of 
192.168.10.61 failed: AUTHENTICATE:1003:Failed to authenticate with any 
method|AUTHENTICATE:1004:Failed to authenticate using KERBEROS
11/12/25 08:17:02 DC_AUTHENTICATE: required authentication of 
192.168.10.61 failed: AUTHENTICATE:1003:Failed to authenticate with any 
method|AUTHENTICATE:1004:Failed to authenticate using KERBEROS

Any ideas would be greatly appreciated as I ran out of them.


___
CIP