[HTCondor-users] Debugging HTCondor Authentication Errors

[Wesley Taylor <wesley.taylor@xxxxxxxxxxx>]

Hello,

I feel a little bad for emailing everyone twice in the same day, but I am still getting familiar with HTCondor.

I tested a minicondor last week and was having a ball of a time, but now I am looking to scale up and have hit some hiccups. I am following the "Setting up an HTCondor Pool" (https://htcondor.readthedocs.io/en/latest/admin-manual/quick-start-condor-pool.html) but with some minor modifications to try and make the test setup better match the production system's architecture. I simply changed the configuration so one machine had the roles of "Submit" and "Central Manager" and I have two "Execute" machines located on the same network. 

I went through the guide, but when I started everything up they weren't authenticating with one another. On both the "Execute" machines I am getting the following from my StartLog (I set STARTD_DEBUG  = D_SECURITY:2 in my config):

__________________________________________________________________________________________________________________________________________________________________________________________________
06/22/20 17:33:14 SECMAN: new session, doing initial authentication.
06/22/20 17:33:14 SECMAN: authenticating RIGHT NOW.
06/22/20 17:33:14 SECMAN: AuthMethodsList: PASSWORD
06/22/20 17:33:14 SECMAN: Auth methods: PASSWORD
06/22/20 17:33:14 AUTHENTICATE: setting timeout for <192.168.0.69:9618> to 20.
06/22/20 17:33:14 AUTHENTICATE: in authenticate( addr == '<192.168.0.69:9618>', methods == 'PASSWORD')
06/22/20 17:33:14 AUTHENTICATE: can still try these methods: PASSWORD
06/22/20 17:33:14 HANDSHAKE: in handshake(my_methods = 'PASSWORD')
06/22/20 17:33:14 HANDSHAKE: handshake() - i am the client
06/22/20 17:33:14 HANDSHAKE: sending (methods == 512) to server
06/22/20 17:33:14 HANDSHAKE: server replied (method = 512)
06/22/20 17:33:14 AUTHENTICATE: will try to use 512 (PASSWORD)
06/22/20 17:33:14 AUTHENTICATE: do_authenticate is 1.
06/22/20 17:33:14 PW.
06/22/20 17:33:14 PW: getting name.
06/22/20 17:33:14 PW: Generating ra.
06/22/20 17:33:14 PW: Client sending.
06/22/20 17:33:14 Client sending: 0, 19(condor_pool@worker1), 256
06/22/20 17:33:14 PW: Client receiving.
06/22/20 17:33:14 Server sent status indicating not OK.
06/22/20 17:33:14 PW: Client received ERROR from server, propagating
06/22/20 17:33:14 PW: CLient sending two.
06/22/20 17:33:14 In client_send_two.
06/22/20 17:33:14 Can't send null for random string.
06/22/20 17:33:14 Client sending: 0() 0 0
06/22/20 17:33:14 Sent ok.
06/22/20 17:33:14 AUTHENTICATE: method 512 (PASSWORD) failed.
06/22/20 17:33:14 AUTHENTICATE: can still try these methods:
06/22/20 17:33:14 HANDSHAKE: in handshake(my_methods = '')
06/22/20 17:33:14 HANDSHAKE: handshake() - i am the client
06/22/20 17:33:14 HANDSHAKE: sending (methods == 0) to server
06/22/20 17:33:14 HANDSHAKE: server replied (method = 0)
06/22/20 17:33:14 AUTHENTICATE: no available authentication methods succeeded!
06/22/20 17:33:14 SECMAN: required authentication with collector 192.168.0.69 failed, so aborting command DC_START_TOKEN_REQUEST.
06/22/20 17:33:14 Failed to request a new token: DAEMON:1:failed to start command for token request with remote daemon at '<192.168.0.69:9618>'.|AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using PASSWORD
__________________________________________________________________________________________________________________________________________________________________________________________________


So then I went and looked at the CollectorLog on the Manager:
__________________________________________________________________________________________________________________________________________________________________________________________________
06/22/20 18:03:15 DC_AUTHENTICATE: required authentication of 192.168.0.70 failed: AUTHENTICATE:1003:Failed to authenticate with any method|AUTHENTICATE:1004:Failed to authenticate using PASSWORD
06/22/20 18:03:15 read_password_from_filename(): read_secure_file(/etc/condor/password.d/POOL) failed!
06/22/20 18:03:15 read_password_from_filename(): read_secure_file(/etc/condor/password.d/POOL) failed!
___________________________________________________________________________________________________________________________________________________________________________________________________

(Don't pay attention to the fact the timestamps are really far apart, I have just been trying some more things in the past little bit)

It looks like for some reason condor can't read the POOL file, even though the file (and its parent directory) are owned by the user and group condor:condor, and everyone has execute permissions on /etc and /etc/condor. I also made selinux permissive just in case that was the issue.

Does anyone have any further steps I can take to figure out why this read is failing? 

Thank you!
-Wes


Wesley Taylor â Cluster Manager
Numerica Corporation (www.numerica.us)
5042 Technology Parkway #100
Fort Collins, Colorado 80528
âï (970) 207 2232
ð wesley.taylor@xxxxxxxxxxx

Attachment: smime.p7s
Description: S/MIME cryptographic signature