[HTCondor-users] Moving CM to new host

[Michael Thomas <wart@xxxxxxxxxxx>]

I'm in the process of moving my 9.0.11 collector/negotiator (CM) from 
SL7.9 to hardware running Rocky 8.  I've copied over the 
$(SPOOL)/Accountantnew.log file as well as the /etc/condor/config.d 
directory.

The one configuration change that was made was to change the hostname 
from 'ldas-condori' to 'ldas-condor.ldas.ligo-la.caltech.edu'.  The IP 
address for the sole network interface was kept the same when 
provisioning the new CM.

After starting up the new collector/negotiator, the collector logs show 
failures communicating with the startds:

04/19/22 12:01:19 DC_AUTHENTICATE: received DC_AUTHENTICATE from 
<10.9.11.223:36338>
04/19/22 12:01:19 Inserting pre-auth metadata for TOKEN.
04/19/22 12:01:19 DC_AUTHENTICATE: Success.
04/19/22 12:01:19 IPVERIFY: for node223.ldas.ligo-la.caltech.edu matched 
10.9.11.223 to 10.9.11.223
04/19/22 12:01:19 PERMISSION DENIED to unauthenticated@unmapped from 
host 10.9.11.223 for command 2 (UPDATE_MASTER_AD), access level 
ADVERTISE_MASTER: reason: ADVERTISE_MASTER authorization policy contains 
no matching ALLOW entry for this request; identifiers used for this 
host: 10.9.11.223,node223.ldas.ligo-la.caltech.edu, hostname size = 1, 
original ip address = 10.9.11.223

This repeats for every startd that tries to connect to the collector.

Current security settings are listed at the end of this message.

The 'DC_AUTHENTICATE: Sucess' followed by 'PERMISSION DENIED to 
unauthenticated@unmapped' seems a bit odd to me.  How can authentication 
succeed but still get mapped to the unauthenticated name?

I verified that the SEC_PASSWORD_FILE had the same contents on both the 
CM and startd.  Then I took a look and found that the 
/etc/condor/tokens.d directory had a single entry on both the CM and the 
startd, which seemed a bit odd.  I would have thought it should be 
filled with tokens for each startd.  I note that this directory also had 
the same single entry on the old CM.

Next I thought that maybe my tokens weren't getting approved, so I ran 
'condor_token_request_auto_approve', but got another authentication failure:

# condor_token_request_auto_approve -netblock 10.13.0.0/16 -lifetime 3600
Failed to create new auto-approval rule: SECMAN:2010:Received "DENIED" 
from server for user unauthenticated@unmapped using no authentication 
method, which may imply host-based security.  Our address was 
'10.13.5.25', and server's address was '10.13.5.25'.  Check your ALLOW 
settings and IP protocols.

Finally, when I tried to fire up my old CM, I started getting the same 
collector errors as on the new CM.

Any suggestions on what to check next to get the collector working again?

--Mike

# condor_config_val -dump SEC_
SEC_C_GAHP_WORKER_THREAD_DEFAULT_SESSION_DURATION = 1800
SEC_CLAIMTOBE_INCLUDE_DOMAIN = false
SEC_CLAIMTOBE_USER =
SEC_CLIENT_AUTHENTICATION = OPTIONAL
SEC_CLIENT_AUTHENTICATION_METHODS = FS, IDTOKENS
SEC_CLIENT_ENCRYPTION = OPTIONAL
SEC_CLIENT_INTEGRITY = OPTIONAL
SEC_CREDENTIAL_REFRESH_INTERVAL = -1
SEC_CREDENTIAL_SWEEP_DELAY = 3600
SEC_CREDENTIAL_SWEEP_INTERVAL = 300
SEC_DEBUG_PRINT_KEYS = false
SEC_DEFAULT_AUTHENTICATION = OPTIONAL
SEC_DEFAULT_AUTHENTICATION_METHODS = FS, IDTOKENS
SEC_DEFAULT_AUTHENTICATION_TIMEOUT = 20
SEC_ENABLE_IMPERSONATION_TOKENS = false
SEC_ENABLE_MATCH_PASSWORD_AUTHENTICATION = true
SEC_IMPERSONATION_TOKEN_LIMITS =
SEC_INVALIDATE_SESSIONS_VIA_TCP = true
SEC_ISSUED_TOKEN_EXPIRATION =
SEC_PASSWORD_DIRECTORY = /etc/condor/passwords.d
SEC_PASSWORD_DOMAIN =
SEC_PASSWORD_FILE = /etc/condor/condor_cred
SEC_READ_ENCRYPTION = OPTIONAL
SEC_READ_INTEGRITY = OPTIONAL
SEC_SCITOKENS_ALLOW_EXTRA_SLASH = false
SEC_SESSION_DURATION_SLOP = 20
SEC_TCP_SESSION_TIMEOUT = 20
SEC_TOKEN_DIRECTORY =
SEC_TOKEN_ISSUER_KEY = POOL
SEC_TOKEN_MAX_AGE =
SEC_TOKEN_POOL_SIGNING_KEY = /etc/condor/condor_cred
SEC_TOKEN_POOL_SIGNING_KEY_FILE = $(SEC_PASSWORD_FILE)
SEC_TOKEN_REQUEST_LIMITS =
SEC_TOKEN_REVOCATION_EXPR =
SEC_TOKEN_SYSTEM_DIRECTORY = /etc/condor/tokens.d
SEC_USE_FAMILY_SESSION = true