Hmm,
MAX_JOBS_SUBMITTED This integer value limits the number of jobs permitted in a condor_schedd daemonâs
queue. Submission of a new cluster of jobs fails, if the total number of jobs would exceed this limit. The default
value for this variable is the largest positive integer value.
This explains also the default in my case ;)
I think at least hold jobs are considered part of the scheds daemon queue (apart from running and idle jobs) and I suspect it might mean roughly all the files in jobqueue.log file which is rotated per default once a day. Hence it contains many more jobs
than you see with condor_q - but that is pure speculation - would explain the behaviour you see though.Â
We use
MAX_JOBS_PER_OWNER This integer value limits the number of jobs any given owner (user) is permitted to have
within a condor_schedd daemonâs queue. A job submission fails if it would cause this limit on the number of
jobs to be exceeded. The default value is 100000.
for the same effect ...
best
christoph
--
Christoph Beyer
DESY Hamburg
IT-Department
Notkestr. 85
Building 02b, Room 009
22607 Hamburg
phone:+49-(0)40-8998-2317
mail:
christoph.beyer@xxxxxxx
Von: "Vikrant Aggarwal" <
ervikrant06@xxxxxxxxx>
An: "HTCondor-Users Mail List" <
htcondor-users@xxxxxxxxxxx>
Gesendet: Mittwoch, 8. MÃrz 2023 16:37:26
Betreff: Re: [HTCondor-users] MAX_JOBS_SUBMITTED exceeded, submit failed. Current total is 499999. Limit is 50000
Hello,Â
Yes, we use it as a protection mechanism so that accident submission of too many jobs didn't kill the schedd. Limit is applicable for all users using the submit box.Â
But when we hit this limit "ERROR" we don't have any job in the queue or sometimes very few which baffled me.
Â
Thanks & Regards,
Vikrant Aggarwal
Hi,
yes I think you are right I was a bit confused as I do not set the value I thought it might be kind of a counter but it seems the default is just set to a very high number ...
[root@bird-htc-sched11 ~]# condor_config_val -v MAX_JOBS_SUBMITTED
MAX_JOBS_SUBMITTED = 2147483647
Â# at: <Default>
Â# raw: MAX_JOBS_SUBMITTED = 2147483647
Is this kind of ceiling for the sched what you need ?
Best
christoph
--
Christoph Beyer
DESY Hamburg
IT-Department
Notkestr. 85
Building 02b, Room 009
22607 Hamburg
phone:+49-(0)40-8998-2317
mail:
christoph.beyer@xxxxxxx
Von: "Vikrant Aggarwal" <
ervikrant06@xxxxxxxxx>
An: "HTCondor-Users Mail List" <
htcondor-users@xxxxxxxxxxx>
Gesendet: Mittwoch, 8. MÃrz 2023 14:54:40
Betreff: Re: [HTCondor-users] MAX_JOBS_SUBMITTED exceeded, submit failed. Current total is 499999. Limit is 50000
Hello,
After making the change to MAX_JOBS_SUBMITTED restarted condor Sched service.Â
# grep restarted /var/log/condor/ScheddRestartReport
The schedd
el6study2.skae.tower-research.com restarted at 03/08/23 05:10:40.
$ condor_config_val MAX_JOBS_SUBMITTED
5
Submitted batch of 5 jobs.Â
$ condor_submit sleep.sub
Submitting job(s).....
5 job(s) submitted to cluster 313.
Trying to submit another batch fails as I have 5 jobs in queue.Â
$ condor_submit sleep.sub
Submitting job(s)
ERROR: Failed to create cluster
Number of submitted jobs would exceed MAX_JOBS_SUBMITTED
If I wait for the completion of existing jobs then I can submit another 5 jobs without any issue which makes me believe that this parameter is related to jobs present in the queue irrespective of their status (hold/running/idle). I don't think it's related
to the total number of jobs submitted in sched..
Thanks & Regards,
Vikrant Aggarwal
Hi,
it seems to me - at least on my scheds the MAX_JOBS_SUBMITTED is indeed the number of jobs the sched dealt with since the last boot (I suppose)
At least this is definetley not the current number of jobs on this sched:
[root@bird-htc-sched11 ~]# condor_config_val MAX_JOBS_SUBMITTED
2147483647
;)
Hence it looks to me as if MAX_JOBS_SUBMITTED should not be set at all unless you want to stop the scheduling after a certain amount of jobs ?
Maybe MAX_JOBS_PER_OWNER is more likely to do what you want (limiting the number of jobs per owner on the sched) ?
Best
christoph
--
Christoph Beyer
DESY Hamburg
IT-Department
Notkestr. 85
Building 02b, Room 009
22607 Hamburg
phone:+49-(0)40-8998-2317
mail:
christoph.beyer@xxxxxxx
Von: "Vikrant Aggarwal" <
ervikrant06@xxxxxxxxx>
An: "HTCondor-Users Mail List" <
htcondor-users@xxxxxxxxxxx>
Gesendet: Mittwoch, 8. MÃrz 2023 08:06:16
Betreff: Re: [HTCondor-users] MAX_JOBS_SUBMITTED exceeded, submit failed. Current total is 499999. Limit is 50000
Thanks Jamie,
But we don't have this many jobs in the queue. The batch we are trying to submit has only a handful of jobs still we are hitting the max job limit.Â
03/07/23 21:46:46 (pid:55697) NewCluster(): MAX_JOBS_SUBMITTED exceeded, submit failed. Current total is 300027. Limit is 300000
03/07/23 22:11:09 (pid:55697) NewCluster(): MAX_JOBS_SUBMITTED exceeded, submit failed. Current total is 300000. Limit is 300000
It's happening randomly but often on a few submit nodes (not all). All submit nodes are with the same conf.ÂÂ
Thanks & Regards,
Vikrant Aggarwal
> On Feb 7, 2023, at 11:01 AM, Todd L Miller via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:
>
>> We hit this issue multiple times: Issue disappears if we restart the condor
>> service or change the MAX_JOBS_SUBMITTED limit.
>
>   ÂYou probably shouldn't be setting MAX_JOBS_SUBMITTED at all. It's a cap on the total number of clusters a schedd is willing to have dealt with for its entire life. What are you trying to accomplish?
This is incorrect. MAX_JOBS_SUBMITTED is a cap on the number of jobs that can be queued at any given time.
Â- Jaime
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to
htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}