Finally able to get the parameters due to which it was happening but didn't understand why it's happening.
IIRC, HTCondor closes (almost?) all FDs after fork()ing* but before exec()ing the shadow. There was not, until relatively recently, a way to close all the FDs associated with a process; you had to make a system call for each FD. When you have to close 102,400 FDs, that's a lot of system calls, and it takes a while.
- ToddM *: On Linux, HTCondor actually calls clone().