[HTCondor-users] Kerberos Permission Denied error

[Asvija B <asvijab@xxxxxxx>]

Dear all,

I am trying to use Kerberos authentication for submitting jobs to HT-Condor. However on the client side the submission fails complaining 'AUTHENTICATE:1002:Failure performing handshake'. The schedd log tells that the permission was denied with this error: DaemonCore: PERMISSION DENIED for 1112 (QMGMT_WRITE_CMD) via TCP from host <10.180.141.148:15918> (access level WRITE)

It is a simple setup to test the Kerberos integration with condor. The KDC is running on the same machine (10.180.141.148).  The same machine has been configured to run as both condor submit node and worker nodes.

I have given the most open options for security in the condor_config file. Following are the excerpts from condor_config file, client debug messages and the schedd log entries:

condor_config file excerpt:

SEC_DEFAULT_NEGOTIATION = OPTIONAL
SEC_DEFAULT_AUTHENTICATION = NEVER
SEC_CLIENT_AUTHENCTICATION = NEVER
SEC_DEFAULT_AUTHENTICATION_METHODS = KERBEROS
KERBEROS_MAP_FILE = $(RELEASE_DIR)/etc/condor.kmap
SCHEDD.ALLOW_WRITE = *@*/*, 10.180.141.148
SEC_WRITE_AUTHENTICATION = NEVER

condor.kmap contents:

[root@gridfs log]# cat /usr/local/nsg/condor/etc/condor.kmap
NSGTEST.CDAC.IN = nsgtest.cdac.in

Kerberos klist output on client side:

[asvija@gridfs condor]$Â klist
Ticket cache: KEYRING:persistent:1005:1005
Default principal: asvija@xxxxxxxxxxxxxxx

Valid startingÂÂÂÂÂÂ ExpiresÂÂÂÂÂÂÂÂÂÂÂÂÂ Service principal
09/06/2019 12:18:30Â 09/07/2019 12:18:30Â krbtgt/NSGTEST.CDAC.IN@xxxxxxxxxxxxxxx

Debug output from condor_submit :

[asvija@gridfs condor]$ _condor_TOOL_DEBUG=D_SECURITY condor_submit -debug condor-universe.job 2>&1 | tee outÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂÂ

09/06/19 12:21:05 KEYCACHE: created: 0x239a150
09/06/19 12:21:05 Can't open directory "/opt/condor//config" as PRIV_UNKNOWN, errno: 2 (No such file or directory)
09/06/19 12:21:05 Cannot open /opt/condor//config: No such file or directory
Submitting job(s)09/06/19 12:21:05 CRED: NO MODULES REQUESTED
09/06/19 12:21:05 SECMAN: command 1112 QMGMT_WRITE_CMD to schedd at <10.180.141.148:9618> from TCP port 22376 (blocking).
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission ALLOW
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission READ
09/06/19 12:21:05 ipverify: READ optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission WRITE
09/06/19 12:21:05 ipverify: WRITE optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission NEGOTIATOR
09/06/19 12:21:05 ipverify: NEGOTIATOR optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission ADMINISTRATOR
09/06/19 12:21:05 ipverify: ADMINISTRATOR optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission OWNER
09/06/19 12:21:05 ipverify: OWNER optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission CONFIG
09/06/19 12:21:05 ipverify: CONFIG optimized to deny everyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission DAEMON
09/06/19 12:21:05 ipverify: DAEMON optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission SOAP
09/06/19 12:21:05 ipverify: SOAP optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission DEFAULT
09/06/19 12:21:05 ipverify: DEFAULT optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission CLIENT
09/06/19 12:21:05 ipverify: CLIENT optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission ADVERTISE_STARTD
09/06/19 12:21:05 ipverify: ADVERTISE_STARTD optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission ADVERTISE_SCHEDD
09/06/19 12:21:05 ipverify: ADVERTISE_SCHEDD optimized to allow anyone
09/06/19 12:21:05 IPVERIFY: Subsystem SUBMIT
09/06/19 12:21:05 IPVERIFY: Permission ADVERTISE_MASTER
09/06/19 12:21:05 ipverify: ADVERTISE_MASTER optimized to allow anyone
09/06/19 12:21:05 AUTHENTICATE: setting timeout for <10.180.141.148:9618?addrs=10.180.141.148-9618&noUDP&sock=83499_42eb_4> to 20.
09/06/19 12:21:05 HANDSHAKE: in handshake(my_methods = 'KERBEROS')
09/06/19 12:21:05 HANDSHAKE: handshake() - i am the client
09/06/19 12:21:05 HANDSHAKE: sending (methods == 64) to server
09/06/19 12:21:05 HANDSHAKE: server replied (method = 64)
09/06/19 12:21:05 KERBEROS: krb5_unparse_name: condor@xxxxxxxxxxxxxxx
09/06/19 12:21:05 KERBEROS: param server princ: condor
09/06/19 12:21:05 KERBEROS: no user yet determined, will grab up to slash
09/06/19 12:21:05 KERBEROS: picked user: condor
09/06/19 12:21:05 Client is condor@xxxxxxxxxxxxxxx
09/06/19 12:21:05 KERBEROS: Server principal is condor@xxxxxxxxxxxxxxx
09/06/19 12:21:05 Acquiring credential for user
09/06/19 12:21:05 Successfully located credential cache
09/06/19 12:21:05 condor_write(): Socket closed when trying to write 13 bytes to schedd at <10.180.141.148:9618>, fd is 4
09/06/19 12:21:05 Buf::write(): condor_write() failed
09/06/19 12:21:05 AUTHENTICATE: method 64 (KERBEROS) failed.
09/06/19 12:21:05 HANDSHAKE: in handshake(my_methods = '')
09/06/19 12:21:05 HANDSHAKE: handshake() - i am the client
09/06/19 12:21:05 HANDSHAKE: sending (methods == 0) to server
09/06/19 12:21:05 condor_write(): Socket closed when trying to write 13 bytes to schedd at <10.180.141.148:9618>, fd is 4
09/06/19 12:21:05 Buf::write(): condor_write() failed
09/06/19 12:21:05 AUTHENTICATE: handshake failed!
09/06/19 12:21:05 Authentication was a FAILURE.

ERROR: Failed to connect to local queue manager
AUTHENTICATE:1002:Failure performing handshake
AUTHENTICATE:1004:Failed to authenticate using KERBEROS

Schedd log:

09/06/19 12:26:22 (pid:83694) ******************************************************
09/06/19 12:26:22 (pid:83694) ** condor_schedd (CONDOR_SCHEDD) STARTING UP
09/06/19 12:26:22 (pid:83694) ** /usr/local/nsg/condor/sbin/condor_schedd
09/06/19 12:26:22 (pid:83694) ** SubsystemInfo: name=SCHEDD type=SCHEDD(5) class=DAEMON(1)
09/06/19 12:26:22 (pid:83694) ** Configuration: subsystem:SCHEDD local:<NONE> class:DAEMON
09/06/19 12:26:22 (pid:83694) ** $CondorVersion: 8.8.4 Jul 09 2019 BuildID: 474941 $
09/06/19 12:26:22 (pid:83694) ** $CondorPlatform: x86_64_RedHat7 $
09/06/19 12:26:22 (pid:83694) ** PID = 83694
09/06/19 12:26:22 (pid:83694) ** Log last touched 9/6 12:26:13
09/06/19 12:26:22 (pid:83694) ******************************************************
09/06/19 12:26:22 (pid:83694) Using config source: /usr/local/nsg/condor/etc/condor_config
09/06/19 12:26:22 (pid:83694) Using local config sources:
09/06/19 12:26:22 (pid:83694)ÂÂÂ /opt/condor//condor_config.local
09/06/19 12:26:22 (pid:83694) config Macros = 99, Sorted = 99, StringBytes = 3606, TablesBytes = 3612
09/06/19 12:26:22 (pid:83694) CLASSAD_CACHING is ENABLED
09/06/19 12:26:22 (pid:83694) Daemon Log is logging: D_ALWAYS D_ERROR
09/06/19 12:26:22 (pid:83694) SharedPortEndpoint: waiting for connections to named socket 83647_c6f0_4
09/06/19 12:26:22 (pid:83694) DaemonCore: command socket at <10.180.141.148:9618?addrs=10.180.141.148-9618&noUDP&sock=83647_c6f0_4>
09/06/19 12:26:22 (pid:83694) DaemonCore: private command socket at <10.180.141.148:9618?addrs=10.180.141.148-9618&noUDP&sock=83647_c6f0_4>
09/06/19 12:26:22 (pid:83694) History file rotation is enabled.
09/06/19 12:26:22 (pid:83694)ÂÂ Maximum history file size is: 20971520 bytes
09/06/19 12:26:22 (pid:83694)ÂÂ Number of rotated history files is: 2
09/06/19 12:26:22 (pid:83694) Reloading job factories
09/06/19 12:26:22 (pid:83694) Loaded 0 job factories, 0 were paused, 0 failed to load
09/06/19 12:26:28 (pid:83694) TransferQueueManager stats: active up=0/100 down=0/100; waiting up=0 down=0; wait time up=0s down=0s
09/06/19 12:26:28 (pid:83694) TransferQueueManager upload 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load
09/06/19 12:26:28 (pid:83694) TransferQueueManager download 1m I/O load: 0 bytes/s 0.000 disk load 0.000 net load
09/06/19 12:27:01 (pid:83694) DaemonCore: PERMISSION DENIED for 1112 (QMGMT_WRITE_CMD) via TCP from host <10.180.141.148:26321> (access level WRITE)

Thanks and regards,

Asvija

------------------------------------------------------------------------------------------------------------
[ C-DAC is on Social-Media too. Kindly follow us at:
Facebook: https://www.facebook.com/CDACINDIA & Twitter: @cdacindia ]

This e-mail is for the sole use of the intended recipient(s) and may
contain confidential and privileged information. If you are not the
intended recipient, please contact the sender by reply e-mail and destroy
all copies and the original message. Any unauthorized review, use,
disclosure, dissemination, forwarding, printing or copying of this email
is strictly prohibited and appropriate legal action will be taken.
------------------------------------------------------------------------------------------------------------