Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Cole,

The cluster master is running on Linux, as are all executor nodes. Submission can occur from Linux and Windows, on Windows jobs run under the user nobody. Therefore thereâs no credd running.

I continued to investigate a bit and took a look at the source code of the authentication. Thereâs one strange thing I donât understand: It should be possible to map usernames via the CERTIFICATE_MAPFILE config variable, which I have tried for testing. However from the debug output it seems as if this map file is never read, which I verified using ProcessMonitor. I built condor on Windows myself (by the way: specifying CMake build type DEBUG still seems to generate a RelWithDebInfo build, at least the output directory had this name) and stepped through the code with a debugger and authenticator_ is always NULL in line 491 of authenticate.cpp, so the map file is never consulted.

Thanks

Martin

From: Cole Bollig <cabollig@xxxxxxxx>
Sent: Monday, June 22, 2026 5:52 PM
To: APEL Martin <Martin.APEL@xxxxxxx>; htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Martin, There should be default mappings to allow NTSSPI map users correctly, so you shouldn't need a map file (there may need to be some other bits). But things are interesting due to the Windows AP into a linux pool. Are the user jobs supposed

Hi Martin,

There should be default mappings to allow NTSSPI map users correctly, so you shouldn't need a map file (there may need to be some other bits). But things are interesting due to the Windows AP into a linux pool. Are the user jobs supposed to run as a particular user on the EP side or can they run as nobody? Just trying to get a big picture idea about your pool setup.

-Cole

From: APEL Martin <Martin.APEL@xxxxxxx>
Sent: Monday, June 22, 2026 9:32 AM
To: Cole Bollig <cabollig@xxxxxxxx>; htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Cole,

Unfortunately this doesnât work either. This is the log with a slightly different error message now:

E:\users\apel>condor_store_cred -d:D_SECURITY add

Account: MAL1@DSONE

CredType: password

Enter password:

06/22/26 16:29:18 STORE_CRED: In mode 100 'add', user is "MAL1@DSONE"

06/22/26 16:29:18 SECMAN: command 479 STORE_CRED to local schedd from TCP port 57767 (blocking).

06/22/26 16:29:18 SECMAN: new session, doing initial authentication.

06/22/26 16:29:18 SECMAN: Auth methods: NTSSPI,TOKEN,PASSWORD

06/22/26 16:29:18 AUTHENTICATE: setting timeout for <192.168.208.34:9618?addrs=192.168.208.34-9618&alias=LP15-MAL1-CEM.dsone.3ds.com&noUDP&sock=schedd_15596_7b7c> to 20.

06/22/26 16:29:18 HANDSHAKE: in handshake(my_methods = 'NTSSPI,TOKEN,PASSWORD')

06/22/26 16:29:18 HANDSHAKE: handshake() - i am the client

06/22/26 16:29:18 HANDSHAKE: sending (methods == 2576) to server

06/22/26 16:29:18 HANDSHAKE: server replied (method = 16)

06/22/26 16:29:18 Authentication was a Success.

06/22/26 16:29:18 AUTHENTICATION: setting default map to (null)

06/22/26 16:29:18 AUTHENTICATION: post-map: current FQU is '(null)'

06/22/26 16:29:18 AUTHENTICATE: Exchanging keys with remote side.

06/22/26 16:29:18 AUTHENTICATE: Result of end of authenticate is 1.

06/22/26 16:29:18 SECMAN: generating AES key for session with local schedd...

06/22/26 16:29:18 SECMAN: successfully enabled encryption!

06/22/26 16:29:18 SECMAN: successfully enabled message authenticator!

06/22/26 16:29:18 SECMAN: FAILED: Received "DENIED" from server for user MAL1.3ds.com@none using method NTSSPI.

06/22/26 16:29:18 ERROR: SECMAN:2010:Received "DENIED" from server for user MAL1.3ds.com@none using method NTSSPI.

06/22/26 16:29:18 STORE_CRED: Failed to start STORE_CRED command. Unable to contact local schedd

Operation failed.

Make sure your ALLOW_WRITE setting includes this host.

I started playing around with the authentication map file, which I previously did not use at all, as my primary auth method is IDTOKEN, which doesnât use this file. However I could not get it to map my Windows username to the one under Linux yet (according to the log output).

Thanks,

Martin

From: Cole Bollig <cabollig@xxxxxxxx>
Sent: Monday, June 22, 2026 4:25 PM
To: APEL Martin <Martin.APEL@xxxxxxx>; htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Martin, Oops, I did mean D_SECURITY. It looks like the client and server (since they share the same configuration) are using TOKEN authentication resulting in the condor user since that is the first method in the list. Try adding NTSSPI to

Hi Martin,

Oops, I did mean D_SECURITY. It looks like the client and server (since they share the same configuration) are using TOKEN authentication resulting in the condor user since that is the first method in the list. Try adding NTSSPI to the start of the list.

-Cole Bollig

From: APEL Martin <Martin.APEL@xxxxxxx>
Sent: Monday, June 22, 2026 9:02 AM
To: Cole Bollig <cabollig@xxxxxxxx>; htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Cole,

You probably meant -debug:D_SECURITY instead of S_SECURITY 😉

This is the generated output:

E:\users\apel>condor_store_cred -debug:D_SECURITY add

Account: MAL1@DSONE

CredType: password

Enter password:

06/22/26 15:59:35 STORE_CRED: In mode 100 'add', user is "MAL1@DSONE"

06/22/26 15:59:35 SECMAN: command 479 STORE_CRED to local schedd from TCP port 61582 (blocking).

06/22/26 15:59:35 SECMAN: new session, doing initial authentication.

06/22/26 15:59:35 SECMAN: Auth methods: TOKEN,NTSSPI,PASSWORD

06/22/26 15:59:35 AUTHENTICATE: setting timeout for <192.168.208.34:9618?addrs=192.168.208.34-9618&alias=LP15-MAL1-CEM.dsone.3ds.com&noUDP&sock=schedd_24784_2504> to 20.

06/22/26 15:59:35 HANDSHAKE: in handshake(my_methods = 'TOKEN,NTSSPI,PASSWORD')

06/22/26 15:59:35 HANDSHAKE: handshake() - i am the client

06/22/26 15:59:35 HANDSHAKE: sending (methods == 2576) to server

06/22/26 15:59:35 HANDSHAKE: server replied (method = 2048)

06/22/26 15:59:35 IDTOKENS: Examining C:\Users\mal1\.condor\tokens.d\mal1.token for valid tokens from issuer none.

06/22/26 15:59:35 Ignoring token as it was signed with key POOL (not known to the server).

06/22/26 15:59:35 getTokenSigningKey(): for id=LOCAL, pool=0 v84mode=0 reading C:\Condor\tokens.sk\LOCAL

06/22/26 15:59:35 Authentication was a Success.

06/22/26 15:59:35 AUTHENTICATION: setting default map to condor@password

06/22/26 15:59:35 AUTHENTICATION: post-map: current FQU is 'condor@password'

06/22/26 15:59:35 AUTHENTICATE: Exchanging keys with remote side.

06/22/26 15:59:35 AUTHENTICATE: Result of end of authenticate is 1.

06/22/26 15:59:35 SECMAN: generating AES key for session with local schedd...

06/22/26 15:59:35 SECMAN: successfully enabled encryption!

06/22/26 15:59:35 SECMAN: successfully enabled message authenticator!

06/22/26 15:59:35 SESSION: client duplicated AES to BLOWFISH key for UDP.

06/22/26 15:59:35 SECMAN: added session LP15-MAL1-CEM:20884:1782136775:4 to cache for 60 seconds (3600s lease).

06/22/26 15:59:35 SECMAN: startCommand succeeded.

Operation failed because it is not allowed

Thanks,

Martin

From: Cole Bollig <cabollig@xxxxxxxx>
Sent: Monday, June 22, 2026 3:48 PM
To: APEL Martin <Martin.APEL@xxxxxxx>; htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Martin, What does the command with -debug:âS_SECURITY say? -Cole From: APEL Martin <Martin.âAPEL@â3ds.âcom> Sent: Monday, June 22, 2026 8:â34 AM To: Cole Bollig <cabollig@âwisc.âedu>; htcondor-users@âcs.âwisc.âedu <htcondor-users@âcs.âwisc.âedu>

Hi Martin,

What does the command with -debug:S_SECURITY say?

-Cole

From: APEL Martin <Martin.APEL@xxxxxxx>
Sent: Monday, June 22, 2026 8:34 AM
To: Cole Bollig <cabollig@xxxxxxxx>; htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Cole,

I have restarted all services on the Windows machine and checked with condor_config_val, that NTSSPI was there.

Thanks,

Martin

From: Cole Bollig <cabollig@xxxxxxxx>
Sent: Monday, June 22, 2026 3:25 PM
To: APEL Martin <Martin.APEL@xxxxxxx>; htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Martin, I believe the authentication is choosing PASSWORD which is POOL password and thus trying as user condor (I do want to admit that the auth code is not my forte). When you added NTSSPI did you do a condor_reconfig? Without a reconfiguration

Hi Martin,

I believe the authentication is choosing PASSWORD which is POOL password and thus trying as user condor (I do want to admit that the auth code is not my forte). When you added NTSSPI did you do a condor_reconfig? Without a reconfiguration the Schedd will not have NTSSPI in its authentication list resulting in the client tool to still use PASSWORD authentication since the server side, i.e. the schedd, will pick the first matching option they both support.

-Cole Bollig

From: APEL Martin <Martin.APEL@xxxxxxx>
Sent: Monday, June 22, 2026 1:54 AM
To: Cole Bollig <cabollig@xxxxxxxx>; htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Cole,

Thanks for the hint, unfortunately it didnât help. What confuses me are these messages in SchedLog:

06/22/26 08:43:56 (pid:13476) WARNING: store_cred() for user username@WinDomainName attempted by user condor, rejecting

It seems as if the interactive invocation of condor_store_cred is carried out as user âcondorâ, not me.

Thanks,

Martin

From: Cole Bollig <cabollig@xxxxxxxx>
Sent: Friday, June 19, 2026 4:42 PM
To: APEL Martin <Martin.APEL@xxxxxxx>; htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Martin, For the windows hosts, try adding NTSSPI to the list of authentication methods (before password). -Cole Bollig From: APEL Martin <Martin.âAPEL@â3ds.âcom> Sent: Friday, June 19, 2026 2:â32 AM To: Cole Bollig <cabollig@âwisc.âedu>;

Hi Martin,

For the windows hosts, try adding NTSSPI to the list of authentication methods (before password).

-Cole Bollig

From: APEL Martin <Martin.APEL@xxxxxxx>
Sent: Friday, June 19, 2026 2:32 AM
To: Cole Bollig <cabollig@xxxxxxxx>; htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Cole,

condor_config_val -dump authentication_methods returns

# Configuration from machine: LP15-MAL1-CEM.dsone.3ds.com

# Parameters with names that match authentication_methods:

SEC_CLIENT_AUTHENTICATION_METHODS = IDTOKENS, PASSWORD

SEC_DEFAULT_AUTHENTICATION_METHODS = IDTOKENS, PASSWORD

SEC_READ_AUTHENTICATION_METHODS = IDTOKENS, PASSWORD

SEC_WRITE_AUTHENTICATION_METHODS = IDTOKENS, PASSWORD

# Contributing configuration file(s):

# C:\Condor\condor_config

# C:\Condor\condor_config.local

I had added the PASSWORDS methods only for Windows, but it seems it doesnât help in any way.

Thanks,

Martin

From: Cole Bollig <cabollig@xxxxxxxx>
Sent: Thursday, June 18, 2026 10:10 PM
To: APEL Martin <Martin.APEL@xxxxxxx>; htcondor-users@xxxxxxxxxxx
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Martin, For the windows submit host what does condor_config_val -dump authentication_methods return? -Cole From: APEL Martin <Martin.âAPEL@â3ds.âcom> Sent: Thursday, June 18, 2026 7:â57 AM To: Cole Bollig <cabollig@âwisc.âedu>; htcondor-users@âcs.âwisc.âedu

Hi Martin,

For the windows submit host what does condor_config_val -dump authentication_methods return?

-Cole

From: APEL Martin <Martin.APEL@xxxxxxx>
Sent: Thursday, June 18, 2026 7:57 AM
To: Cole Bollig <cabollig@xxxxxxxx>; htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: RE: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Cole,

Thank you for your quick response. Regarding your questions:

1. I run condor_store_cred add without any additional parameters

2. Adding the -debug:D:HOST_NAME generates the following output (I have replaced domain and usernames):

condor_store_cred -debug:D_HOSTNAME add

06/18/26 08:29:16 NETWORK_INTERFACE=* matches Ethernet fe80::d7db:b5ca:797a:9984, Ethernet 169.254.55.6, Ethernet 4 fe80::f680:c67b:aff5:9b, Ethernet 4 169.254.71.157, Ethernet 2 fe80::2328:e55a:26cb:dadd, Ethernet 2 192.168.208.34, Wi-Fi fe80::f224:7aa9:83bb:1dc2, Wi-Fi 169.254.98.211, Local Area Connection* 1 fe80::894c:4326:1ed2:e343, Local Area Connection* 1 169.254.118.174, Local Area Connection* 12 fe80::b59:9f8e:7b22:bbff, Local Area Connection* 12 169.254.172.248, Bluetooth Network Connection fe80::d6c5:e04c:b18:7898, Bluetooth Network Connection 169.254.89.57, Loopback Pseudo-Interface 1 ::1, Loopback Pseudo-Interface 1 127.0.0.1, vEthernet (nat) fe80::660:43c2:243:4ea, vEthernet (nat) 192.168.224.1, choosing IP 192.168.208.34

06/18/26 08:29:16 hostname: hostname.dnsdomainname

06/18/26 08:29:16 I am: hostname: hostname, fully qualified doman name: hostname.dnsdomainname, IP: 192.168.208.34, IPv4: 192.168.208.34, IPv6:

06/18/26 08:29:16 Trying to getting network interface information after reading config

06/18/26 08:29:16 hostname: hostname.dnsdomainname

06/18/26 08:29:16 I am: hostname: hostname, fully qualified doman name: hostname.dnsdomainname, IP: 192.168.208.34, IPv4: 192.168.208.34, IPv6:

Account: username@WinDomainName

CredType: password

Enter password:

06/18/26 08:29:22 STORE_CRED: In mode 100 'add', user is "username@WinDomainName"

06/18/26 08:29:22 New Daemon obj (schedd) name: "", pool: "", addr: ""

06/18/26 08:29:22 Neither name nor addr specified, using local values - name: "hostname.dnsdomainname", full host: "hostname.dnsdomainname"

06/18/26 08:29:22 Finding classad for local daemon, SCHEDD_DAEMON_AD_FILE is "C:\Condor\spool/.schedd_classad"

06/18/26 08:29:22 Found Name in ClassAd, using "hostname.dnsdomainname"

06/18/26 08:29:22 Daemon client (schedd) address determined: name: "hostname.dnsdomainname", pool: "", alias: "hostname.dnsdomainname", addr: "<192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4>"

06/18/26 08:29:22 Found SCHEDDIpAddr in ClassAd, using "<192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4>"

06/18/26 08:29:22 Found CondorVersion in ClassAd, using "$CondorVersion: 25.11.0 2026-06-10 BuildID: 920473 GitSHA: 7f5259d9 $"

06/18/26 08:29:22 Found CondorPlatform in ClassAd, using "$CondorPlatform: x86_64_Windows10 $"

06/18/26 08:29:22 Found Machine in ClassAd, using "hostname.dnsdomainname"

06/18/26 08:29:22 Checking if <192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4> is a sinful address

06/18/26 08:29:22 <192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4> is a sinful address!

06/18/26 08:29:22 Using port 9618 based on address "<192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4>"

06/18/26 08:29:22 Found address 1 candidates:

06/18/26 08:29:22 -410 192.168.208.34:9618

06/18/26 08:29:22 Considering address candidate 192.168.208.34:9618.

06/18/26 08:29:22 Found compatible candidate 192.168.208.34:9618.

06/18/26 08:29:22 Destroying Daemon object:

06/18/26 08:29:22 Type: 3 (schedd), Name: hostname.dnsdomainname, Addr: <192.168.208.34:9618?addrs=192.168.208.34-9618&alias=hostname.dnsdomainname&noUDP&sock=schedd_6452_7bd4>

06/18/26 08:29:22 FullHost: hostname.dnsdomainname, Host: hostname, Pool: , Port: 9618

06/18/26 08:29:22 IsLocal: Y, IdStr: local schedd, Error:

06/18/26 08:29:22 --- End of Daemon object info ---

Operation failed because it is not allowed

3. The master node as well as all execution nodes of the cluster run on Linux. Submit hosts are both Linux and Windows. On Windows I do not use the ârun_as_ownerâ feature.

Thanks,

Martin

From: Cole Bollig <cabollig@xxxxxxxx>
Sent: Wednesday, June 17, 2026 5:26 PM
To: htcondor-users@xxxxxxxxxxx
Cc: APEL Martin <Martin.APEL@xxxxxxx>
Subject: Re: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

Hi Martin, Some questions to help look into this: What specifically are you running on the command line to store credentials? Can you run the same command with the -debug:âD_HOSTNAME option and share the resulting output (feel free to cleanse

Hi Martin,

Some questions to help look into this:

1. What specifically are you running on the command line to store credentials?

2. Can you run the same command with the -debug:D_HOSTNAME option and share the resulting output (feel free to cleanse and/or send directly)?

3. You mentioned having a mixed pool. What OS are the Aps (submit hosts)?

Cheers,

Cole Bollig

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of APEL Martin via HTCondor-users <htcondor-users@xxxxxxxxxxx>
Sent: Wednesday, June 17, 2026 5:17 AM
To: htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Cc: APEL Martin <Martin.APEL@xxxxxxx>
Subject: [HTCondor-users] Trouble with authentication on Windows after upgrade from 8.9 to 25.11

ZjQcmQRYFpfptBannerEnd

We have recently upgraded our HTCondor cluster from 8.9 to 25.11. The cluster contains Linux as well as Windows machines. All authentication and authorization is configured to use IDTOKENS, which works fine under Linux. However when using the same approach on Windows any submissions fail and tell me, that I need to use condor_store_cred.

When I run condor_store_cred add I get an error âOperation failed because it is not allowedâ after entering the password.

SchedLog contains entries such as

06/17/26 11:58:38 (pid:25592) WARNING: store_cred() for user user@domain attempted by user condor, rejecting

I have to add that the DNS domain is not identical to the Windows domain name. I have tried adding both domains to the âALLOW_*â configurations, I have tried enabling PASSWORD authentication, but nothing seems to help. I have a token for user@winDomain in my tokens.d directory, which allows me to run e.g. condor_status. I tried the same with a token for user@dnsdomain, condor_status works here as well.

But condor_store_cred continues to fail in all these cases.

Any help would be very much appreciated.

Martin

This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.

If you are not one of the named recipients or have received this email in error,

(i) you should not read, disclose, or copy it,

(ii) please notify sender of your receipt by reply email and delete this email and all attachments,