Re: [HTCondor-users] how to knock at shared port daemon for initiating/debugging SSL handshake

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi Brian,

thanks for the info

I think we have identified at least a part of the issue - which seems to
relate to the structure of the hosts' certificate bag structure.

On Friday noon, we deployed  new host certificates on our production
CEs, where the host certificate pem file consisted of the actual host
certificate followed by two BEGIN/END CERTIFICATE blocks with the
issuers certs.

Since then handshakes failed for the Belle2 group (trying to gather
their SSL libs in use) and partly locally [1]. After I now truncated the
cert bag structs to the host cert only removing the tailing issuer
certs, the handshakes became successful again.

However, during the period other users however other groups like ATLAS,
CMS, LHCb (where LHCb should be in principle using the same framework as
Belle2) saw no problems and handshaked as usual with the hostcert+issuer
bag struct.
I.e., some SSL libraries(?)/implementations seem to work while others
fail - the condor and ssl packages on the CE as well as our local client
look like [2], where both are CentOS 7.9.2009.

Cheers,
   Thomas

[1]
It failed at us for for condor_ce_{trace,ping} with X509 authz for proxy
certs wfrom the same CA as the CE's host cert - while it worked when
using grid proxy cert originating from another CA.


[2]
[2.CondorCE]
htcondor-ce-bdii-5.1.3-1.el7.noarch
condor-classads-9.0.11-1.el7.x86_64
htcondor-ce-client-5.1.3-1.el7.noarch
python3-condor-9.0.11-1.el7.x86_64
htcondor-ce-view-5.1.3-1.el7.noarch
python2-condor-9.0.11-1.el7.x86_64
htcondor-ce-condor-5.1.3-1.el7.noarch
condor-procd-9.0.11-1.el7.x86_64
condor-externals-9.0.11-1.el7.x86_64
condor-boinc-7.16.16-1.el7.x86_64
htcondor-ce-5.1.3-1.el7.noarch
condor-9.0.11-1.el7.x86_64
htcondor-ce-apel-5.1.3-1.el7.noarch
globus-gsi-openssl-error-4.3-1.el7.x86_64
globus-gsi-proxy-ssl-6.5-1.el7.x86_64
globus-openssl-module-5.2-1.el7.x86_64
openssl-1.0.2k-25.el7_9.x86_64
openssl-libs-1.0.2k-25.el7_9.x86_64
perl-Crypt-OpenSSL-X509-1.803-4.el7.x86_64
perl-IO-Socket-SSL-1.94-7.el7.noarch
perl-Net-SSLeay-1.55-6.el7.x86_64
python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch


[2.Client]
condor-9.0.11-1.el7.x86_64
condor-classads-9.0.11-1.el7.x86_64
condor-externals-9.0.11-1.el7.x86_64
condor-procd-9.0.11-1.el7.x86_64
htcondor-ce-client-5.1.3-1.el7.noarch
python2-condor-9.0.11-1.el7.x86_64
python3-condor-9.0.11-1.el7.x86_64
globus-gsi-openssl-error-4.3-1.el7.x86_64
globus-gsi-proxy-ssl-6.5-1.el7.x86_64
globus-openssl-module-5.2-1.el7.x86_64
gskssl64-8.0-55.24.x86_64
mod_ssl-2.4.6-97.el7.centos.5.x86_64
openssl098e-0.9.8e-29.el7.centos.3.x86_64
openssl-1.0.2k-25.el7_9.x86_64
openssl-devel-1.0.2k-25.el7_9.x86_64
openssl-libs-1.0.2k-25.el7_9.x86_64
perl-Crypt-OpenSSL-X509-1.803-4.el7.x86_64
perl-IO-Socket-SSL-1.94-7.el7.noarch
perl-Net-SSLeay-1.55-6.el7.x86_64
python-backports-ssl_match_hostname-3.5.0.1-1.el7.noarch



On 11/04/2022 15.50, Bockelman, Brian wrote:
> Hi Thomas,
> 
> Unfortunately, there's no simple way to debug - the TLS handshake occurs in the middle of HTCondor's binary protocol.
> 
> I usually debug such things by setting TOOL_DEBUG=D_FULLDEBUG,D_SECURITY.  At that level, the client does a reasonably good job of at least emitting the OpenSSL error messages -- and debug from there.
> 
> Brian
> 
>> On Apr 11, 2022, at 6:54 AM, Thomas Hartmann <thomas.hartmann@xxxxxxx> wrote:
>>
>> Hi all,
>>
>> is there a way to connect with `openssl s_client` to a CondorCE running
>> a shared port daemon?
>>
>> I.e., I would like to debug a probable certificate issue for one of our
>> VOs, where their connection fails early - and my suspicion is, that
>> their trusted CA chain is not in order. Thus, I would like them just to
>> initiate the SSL handshake for more details.
>> However, no SSL handshake is initiated when going directly for the
>> CE:port - as I suppose that one needs to knock correctly for the shared
>> port daemon, or?
>>
>> Cheers,
>>  Thomas
>> _______________________________________________
>> HTCondor-users mailing list
>> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/htcondor-users/
> 
> 
> _______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature