Mailing List Archives

Authenticated access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HTCondor-users] condor_rm job, Permission denied to force removal

Hi John,
Thanks for the help. I have attached SchedLog output after adding SCHEDD_DEBUG = $(SCHEDD_DEBUG) D_SECURITY:2 and running the command,Â
condor_rm 2.0

Could this be a upper / lower case issue?ÂÂ

One thing I see in the log in this,
AuthenticatedName = "ajbarr@COMPANY"

But after that I see this,
User = "ajbarr@company"

Note, I sanitized the log and replaced the hostnames with general names.Â

Thanks,
Andy





On Tue, Jan 21, 2025 at 5:54âPM John M Knoeller <johnkn@xxxxxxxxxxx> wrote:
01/21/25 13:15:01 HANDSHAKE: in handshake(my_methods = 'NTSSPI,PASSWORD')
01/21/25 13:15:01 HANDSHAKE: handshake() - i am the client
01/21/25 13:15:01 HANDSHAKE: sending (methods == 528) to server
01/21/25 13:15:01 HANDSHAKE: server replied (method = 16)
01/21/25 13:15:01 Authentication was a Success.
01/21/25 13:15:01 AUTHENTICATION: setting default map to (null)
01/21/25 13:15:01 AUTHENTICATION: post-map: current FQU is '(null)'

This shows that NTSSPI (method bit 16) was the authentication method used, but for some reason
the authenticated identity could not be converted to a username.  (FQU is fully qualified user. )

I think we need to see the SchedLog, try adding this to your configuration, then reconfig the sched and reproduce the problem.Â

SCHEDD_DEBUG = $(SCHEDD_DEBUG) D_SECURITY:2

This will produce a lot of output in the SchedLog, but I think we need the detailed logging to give us some clue why NTSSPI authentication is succeeding, but the username ends up being anonymous anyway.

-tj

From:ÂAndy Barr <ajbarr@xxxxxxxxx>
Sent:ÂTuesday, January 21, 2025 2:56 PM
To:ÂHTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Cc:ÂJohn M Knoeller <johnkn@xxxxxxxxxxx>
Subject:ÂRe: [HTCondor-users] condor_rm job, Permission denied to force removal
Â
Hi John,
What Condor version? Â
$CondorVersion: 24.2.2 2024-12-04 BuildID: 772905 GitSHA: 2b56256d $
$CondorPlatform: x86_64_Windows10 $

Can you submit new jobs to the schedd?Â
Yes

Are you logged in to the machine that the SCHEDD is running on? or are you trying to remove jobs from a SCHEDD remotely? ÂSome authorization methods only work locally.ÂÂ
Yes, but I would like to be able to remove jobs from a SCHEDD remotely eventually.

If you are running Condor version 24 or later, you can try

condor_rm 2.0 -debug:D_SECURITY
01/21/25 13:15:01 Win32 sysapi_get_network_device_info_raw()
01/21/25 13:15:01 SECMAN: command 478 ACT_ON_JOBS to <10.29.4.45:9618> from TCP port 49801 (blocking).
01/21/25 13:15:01 SECMAN: new session, doing initial authentication.
01/21/25 13:15:01 SECMAN: Auth methods: NTSSPI,PASSWORD
01/21/25 13:15:01 AUTHENTICATE: setting timeout for <10.29.4.45:9618?addrs=10.29.4.45-9618&alias=node1.company.com&noUDP&sock=schedd_15316_70f8> to 20.
01/21/25 13:15:01 HANDSHAKE: in handshake(my_methods = 'NTSSPI,PASSWORD')
01/21/25 13:15:01 HANDSHAKE: handshake() - i am the client
01/21/25 13:15:01 HANDSHAKE: sending (methods == 528) to server
01/21/25 13:15:01 HANDSHAKE: server replied (method = 16)
01/21/25 13:15:01 Authentication was a Success.
01/21/25 13:15:01 AUTHENTICATION: setting default map to (null)
01/21/25 13:15:01 AUTHENTICATION: post-map: current FQU is '(null)'
01/21/25 13:15:01 AUTHENTICATE: Exchanging keys with remote side.
01/21/25 13:15:01 AUTHENTICATE: Result of end of authenticate is 1.
01/21/25 13:15:01 SECMAN: generating AES key for session with <10.29.4.45:9618>...
01/21/25 13:15:01 SECMAN: successfully enabled encryption!
01/21/25 13:15:01 SECMAN: successfully enabled message authenticator!
01/21/25 13:15:01 SESSION: client duplicated AES to BLOWFISH key for UDP.
01/21/25 13:15:01 SECMAN: added session P01200537:17268:1737483301:11 to cache for 60 seconds (3600s lease).
01/21/25 13:15:01 SECMAN: startCommand succeeded.
01/21/25 13:15:01 DCSchedd:actOnJobs: Action failed

On Tue, Jan 21, 2025 at 12:21âPM John M Knoeller via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:
What Condor version? Â

Can you submit new jobs to the schedd?Â

Are you logged in to the machine that the SCHEDD is running on? or are you trying to remove jobs from a SCHEDD remotely? ÂSome authorization methods only work locally.ÂÂ

If you are running Condor version 24 or later, you can try

condor_rm 24.0 -debug:D_SECURITYÂ

To get more detailed logging, but we probably need D_SECURITY logging from the SchedLog to see why it is not authenticating you.Â

-tj


From:ÂHTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Andy Barr <ajbarr@xxxxxxxxx>
Sent:ÂSunday, January 19, 2025 8:02 AM
To: htcondor-users@xxxxxxxxxxxÂ<htcondor-users@xxxxxxxxxxx>
Subject:Â[HTCondor-users] condor_rm job, Permission denied to force removal
Â
Hi,
I'm trying to remove jobs that are in the HOLD state in my condor pool. This is a small windows OS only pool that I am working on setting up. I am the owner of the job

OWNER Â BATCH_NAME Â ÂSUBMITTED Â DONE Â RUN Â ÂIDLE Â HOLD ÂTOTAL JOB_IDS
ajbarr ID: 24 Â Â Â12/13 17:18 Â Â Â_ Â Â Â_ Â Â Â_ Â Â Â1 Â Â Â1 24.0

I'm using the command,

condor_rm -force 24.0
Permission denied to force removal of job 24.0

Last, I get this error message in my SchedLog,
01/19/25 08:57:47 (pid:27872) QMGT command failed: anonymous user not permitted

so it seems for some reason it thinks I'm an anonymous user?
from a dos prompt I get,
whoami
company\ajbarr

I am able to successfully run jobs on this pool.Â

Thanks forÂyour help,
Andy
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxxÂwith a
subject: Unsubscribe

The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/
01/22/25 08:08:06 (pid:16332) DaemonCommandProtocol: Not enough bytes are ready for read.
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: received DC_AUTHENTICATE from <10.29.4.45:54102>
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: received following ClassAd:
AuthMethods = "NTSSPI,PASSWORD"
Authentication = "REQUIRED"
AuthenticationNew = "REQUIRED"
Command = 478
ConnectSinful = "<10.29.4.45:9618?addrs=10.29.4.45-9618&alias=node1.company.com&noUDP&sock=schedd_7472_c6b6>"
CryptoMethods = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BBrr1yHRSk7C7uHJucGwc+StWXff74hBB5NWWaeUlGJ9sWdKOs+65KuxiC3dryq8KcHB+HZE4pcT1BiQBurH5t8="
Enact = "NO"
Encryption = "REQUIRED"
Integrity = "REQUIRED"
NegotiatedSession = true
NewSession = "YES"
OutgoingNegotiation = "REQUIRED"
RemoteVersion = "$CondorVersion: 24.2.2 2024-12-04 BuildID: 772905 GitSHA: 2b56256d $"
ServerPid = 11904
SessionDuration = "60"
SessionLease = 3600
Subsystem = "TOOL"
TrustDomain = "node1.company.com"
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: our_policy:
AuthMethods = "NTSSPI,TOKEN,PASSWORD,KERBEROS"
Authentication = "REQUIRED"
AuthenticationNew = "REQUIRED"
CryptoMethods = "AES,BLOWFISH,3DES"
Enact = "NO"
Encryption = "REQUIRED"
Integrity = "REQUIRED"
IssuerKeys = "LOCAL, POOL"
OutgoingNegotiation = "REQUIRED"
ParentUniqueID = "node1:7472:1737551184"
ServerPid = 16332
SessionDuration = "86400"
SessionLease = 3600
Subsystem = "SCHEDD"
TrustDomain = "node1.company.com"
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: the_policy:
AuthMethods = "NTSSPI"
AuthMethodsList = "NTSSPI,PASSWORD"
Authentication = "YES"
CryptoMethods = "AES,BLOWFISH,3DES"
CryptoMethodsList = "AES,BLOWFISH,3DES"
Enact = "YES"
Encryption = "YES"
Integrity = "YES"
IssuerKeys = "LOCAL, POOL"
SessionDuration = "60"
SessionLease = 3600
TrustDomain = "node1.company.com"
01/22/25 08:08:06 (pid:16332) SECMAN: Sending following response ClassAd:
AuthMethods = "NTSSPI"
AuthMethodsList = "NTSSPI,PASSWORD"
Authentication = "YES"
CryptoMethods = "AES"
CryptoMethodsList = "AES,BLOWFISH,3DES"
ECDHPublicKey = "BO4IJWW/8Bf3o6OwmvrQ//kNVZEADE5wEMVnZ/jy/QuQ09BiEM47hg0BiEXYzhC/9Ii8qiXWiSAGZZcYNUo+6Cw="
Enact = "YES"
Encryption = "YES"
Integrity = "YES"
IssuerKeys = "LOCAL, POOL"
NegotiatedSession = true
RemoteVersion = "$CondorVersion: 24.2.2 2024-12-04 BuildID: 772905 GitSHA: 2b56256d $"
SessionDuration = "60"
SessionLease = 3600
TrustDomain = "node1.company.com"
01/22/25 08:08:06 (pid:16332) SECMAN: new session, doing initial authentication.
01/22/25 08:08:06 (pid:16332) Returning to DC while we wait for socket to authenticate.
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: authenticating RIGHT NOW.
01/22/25 08:08:06 (pid:16332) AUTHENTICATE: setting timeout for (unknown) to 20.
01/22/25 08:08:06 (pid:16332) AUTHENTICATE: in authenticate( addr == '(unknown)', methods == 'NTSSPI,PASSWORD')
01/22/25 08:08:06 (pid:16332) AUTHENTICATE: can still try these methods: NTSSPI,PASSWORD
01/22/25 08:08:06 (pid:16332) HANDSHAKE: in handshake(my_methods = 'NTSSPI,PASSWORD')
01/22/25 08:08:06 (pid:16332) HANDSHAKE: handshake() - i am the server
01/22/25 08:08:06 (pid:16332) HANDSHAKE: client sent (methods == 528)
01/22/25 08:08:06 (pid:16332) HANDSHAKE: i picked (method == 16)
01/22/25 08:08:06 (pid:16332) HANDSHAKE: client received (method == 16)
01/22/25 08:08:06 (pid:16332) AUTHENTICATE: will try to use 16 (NTSSPI)
01/22/25 08:08:06 (pid:16332) AUTHENTICATE: do_authenticate is 1.
01/22/25 08:08:06 (pid:16332) AUTHENTICATE: auth_status == 16 (NTSSPI)
01/22/25 08:08:06 (pid:16332) Authentication was a Success.
01/22/25 08:08:06 (pid:16332) AUTHENTICATION: setting default map to ajbarr@company
01/22/25 08:08:06 (pid:16332) AUTHENTICATION: post-map: current user is 'ajbarr'
01/22/25 08:08:06 (pid:16332) AUTHENTICATION: post-map: current domain is 'company'
01/22/25 08:08:06 (pid:16332) AUTHENTICATION: post-map: current FQU is 'ajbarr@company'
01/22/25 08:08:06 (pid:16332) AUTHENTICATE: Exchanging keys with remote side.
01/22/25 08:08:06 (pid:16332) AUTHENTICATE: Result of end of authenticate is 1.
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: authentication of 10.29.4.45 complete.
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: generating AES key for session node1:16332:1737551286:1...
01/22/25 08:08:06 (pid:16332) CRYPTO: New crypto state with protocol AES
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: encryption enabled for session node1:16332:1737551286:1
01/22/25 08:08:06 (pid:16332) SECMAN: because protocal is AES, not using other MAC.
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: message authenticator enabled with key id node1:16332:1737551286:1.
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: Success.
01/22/25 08:08:06 (pid:16332) PERMISSION GRANTED to ajbarr@company from host 10.29.4.45 for command 478 (ACT_ON_JOBS), access level WRITE: reason: cached result for WRITE; see first case for the full reason
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: sending session ad:
ReturnCode = "AUTHORIZED"
Sid = "node1:16332:1737551286:1"
TriedAuthentication = true
User = "ajbarr@company"
ValidCommands = "60021,60052,421,478,480,486,488,489,487,499,531,464,479,541,542,1112,509,511,526,527,528,521,507,60007,457,60020,443,441,6,12,5,515,516,519,540,1111"
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: sent session node1:16332:1737551286:1 info!
01/22/25 08:08:06 (pid:16332) SESSION: fallback crypto method would be BLOWFISH.
01/22/25 08:08:06 (pid:16332) SESSION: server checking key type: 3
01/22/25 08:08:06 (pid:16332) SESSION: found list: AES,BLOWFISH,3DES.
01/22/25 08:08:06 (pid:16332) SESSION: server duplicated AES to BLOWFISH key for UDP.
01/22/25 08:08:06 (pid:16332) DC_AUTHENTICATE: added incoming session id node1:16332:1737551286:1 to cache for 80 seconds (lease is 3620s, return address is ).
AuthMethods = "NTSSPI"
AuthMethodsList = "NTSSPI,PASSWORD"
AuthenticatedName = "ajbarr@COMPANY"
Authentication = "YES"
CryptoMethods = "AES"
CryptoMethodsList = "AES,BLOWFISH,3DES"
Enact = "YES"
Encryption = "YES"
Integrity = "YES"
IssuerKeys = "LOCAL, POOL"
NegotiatedSession = true
RemoteVersion = "$CondorVersion: 24.2.2 2024-12-04 BuildID: 772905 GitSHA: 2b56256d $"
ServerPid = 11904
SessionDuration = "60"
SessionLease = 3600
Sid = "node1:16332:1737551286:1"
Subsystem = "TOOL"
TriedAuthentication = true
TrustDomain = "node1.company.com"
User = "ajbarr@company"
ValidCommands = "60021,60052,421,478,480,486,488,489,487,499,531,464,479,541,542,1112,509,511,526,527,528,521,507,60007,457,60020,443,441,6,12,5,515,516,519,540,1111"
01/22/25 08:08:06 (pid:16332) QMGT command failed: anonymous user not permitted
01/22/25 08:08:26 (pid:16332) Activity on stashed negotiator socket: <10.29.4.45:54097>
01/22/25 08:08:26 (pid:16332) Negotiating for owner: ajbarr@xxxxxxxxxxxxxxxxx
01/22/25 08:08:26 (pid:16332) Finished sending rrls to negotiator
01/22/25 08:08:26 (pid:16332) Finished sending RRL for ajbarr
01/22/25 08:08:26 (pid:16332) Activity on stashed negotiator socket: <10.29.4.45:54097>
01/22/25 08:08:26 (pid:16332) Negotiating for owner: ajbarr@xxxxxxxxxxxxxxxxx
01/22/25 08:08:26 (pid:16332) Negotiation ended: 0 jobs matched
01/22/25 08:08:26 (pid:16332) Finished negotiating for ajbarr in local pool: 0 matched, 1 rejected
01/22/25 08:09:26 (pid:16332) Activity on stashed negotiator socket: <10.29.4.45:54097>
01/22/25 08:09:26 (pid:16332) Negotiating for owner: ajbarr@xxxxxxxxxxxxxxxxx
01/22/25 08:09:26 (pid:16332) Finished sending rrls to negotiator
01/22/25 08:09:26 (pid:16332) Finished sending RRL for ajbarr
01/22/25 08:09:26 (pid:16332) Activity on stashed negotiator socket: <10.29.4.45:54097>
01/22/25 08:09:26 (pid:16332) Negotiating for owner: ajbarr@xxxxxxxxxxxxxxxxx
01/22/25 08:09:26 (pid:16332) Negotiation ended: 0 jobs matched
01/22/25 08:09:26 (pid:16332) Finished negotiating for ajbarr in local pool: 0 matched, 1 rejected