Thanks Jamie,
But we don't have this many jobs in the queue. The batch we are trying to submit has only a handful of jobs still we are hitting the max job limit.Â
03/07/23 21:46:46 (pid:55697) NewCluster(): MAX_JOBS_SUBMITTED exceeded, submit failed. Current total is 300027. Limit is 300000
03/07/23 22:11:09 (pid:55697) NewCluster(): MAX_JOBS_SUBMITTED exceeded, submit failed. Current total is 300000. Limit is 300000
It's happening randomly but often on a few submit nodes (not all). All submit nodes are with the same conf.ÂÂ
Thanks & Regards,
Vikrant Aggarwal