[HTCondor-users] CE Schedd segfaults during auth (?)

["Fischer, Max (SCC)" <max.fischer@xxxxxxx>]

Hi all,

Today one of our HTCondor-CE Schedds has started regularly segfaulting [0] during what appears to be authentication. The log messages right before the segfault seem unrelated, and we see multiple successful SCITOKENS and SSL auths before the issue occurs. The issue occurs pretty frequently, with the Schedd surviving for about 1 minute at most.
The machine is running HTCondor 23.7.2. Two other machines are setup similarly and should run jobs from the same submitters, yet do not show the problem.

Neither restarting the Schedd, the entire condor service nor rebooting the machine was enough to fix the issue.

Any idea what may be causing this?

Cheers,
Max

[0]
Caught signal 6: si_code=4294967290, si_pid=2510340, si_uid=0, si_addr=0x264E04
Stack dump for process 2510340 at timestamp 1717686522 (19 frames)
/lib64/libcondor_utils_23_7_2.so(_Z18dprintf_dump_stackv+0x28)[0x7f3a568c54a8]
/lib64/libcondor_utils_23_7_2.so(_Z17unix_sig_coredumpiP9siginfo_tPv+0x6c)[0x7f3a56ac143c]
/lib64/libpthread.so.0(+0x12cf0)[0x7f3a54b23cf0]
/lib64/libc.so.6(gsignal+0x10f)[0x7f3a5479aacf]
/lib64/libc.so.6(abort+0x127)[0x7f3a5476dea5]
/lib64/libclassad.so.23.7.2(+0xa5c60)[0x7f3a56e80c60]
/lib64/libcondor_utils_23_7_2.so(_ZN15Condor_Auth_SSL28authenticate_server_scitokenEP11CondorErrorb+0x6ef)[0x7f3a56a1e02f]
/lib64/libcondor_utils_23_7_2.so(_ZN14Authentication21authenticate_continueEP11CondorErrorb+0xc6c)[0x7f3a569f9b7c]
/lib64/libcondor_utils_23_7_2.so(_ZN8ReliSock21authenticate_continueEP11CondorErrorbPPc+0x2a)[0x7f3a56a4471a]
/lib64/libcondor_utils_23_7_2.so(_ZN21DaemonCommandProtocol20AuthenticateContinueEv+0x46)[0x7f3a56a9a9d6]
/lib64/libcondor_utils_23_7_2.so(_ZN21DaemonCommandProtocol10doProtocolEv+0xe5)[0x7f3a56aa0f15]
/lib64/libcondor_utils_23_7_2.so(_ZN21DaemonCommandProtocol14SocketCallbackEP6Stream+0xa3)[0x7f3a56aa1083]
/lib64/libcondor_utils_23_7_2.so(_ZN10DaemonCore24CallSocketHandler_workerEibP6Stream+0x1e0)[0x7f3a56aa9180]
/lib64/libcondor_utils_23_7_2.so(_ZN10DaemonCore35CallSocketHandler_worker_demarshallEPv+0x21)[0x7f3a56aa93c1]
/lib64/libcondor_utils_23_7_2.so(_ZN13CondorThreads8pool_addEPFvPvES0_PiPKc+0x3c)[0x7f3a5686fd9c]
/lib64/libcondor_utils_23_7_2.so(_ZN10DaemonCore6DriverEv+0xdce)[0x7f3a56aad3be]
/lib64/libcondor_utils_23_7_2.so(_Z7dc_mainiPPc+0x1787)[0x7f3a56acb387]
/lib64/libc.so.6(__libc_start_main+0xe5)[0x7f3a54786d85]

Attachment: smime.p7s
Description: S/MIME cryptographic signature