|
Hi Stefano,
Your recollection is correct to the best of my knowledge --
At the time, the condor_schedd process used about 40KB of memory per idle job and, given the number of DAGs running per condor_sched multiplied by the size of the DAG, there simply wasn't enough memory.
I also suspect that we utilized max-idle to reduce the load on the condor_schedd generated by the submission of many large DAGs simultaneously: the "job submission Hz" is a finite resource and max-idle allows one to smooth out the bursts.
Thanks,
Brian
On Jan 7, 2025, at 2:02âAM, Stefano Belforte via HTCondor-users <htcondor-users@xxxxxxxxxxx> wrote:
Hi Cole,
we set that in CMS CRAB since ever (~10y) Maybe at the time there was a concern that too many idle jobs were bad for HTC (possibly autoclusters were not there yet ? only Brian B. could tell, if he remembers). At the moment it is simply "it works, don't
fix".
Stefano
On 06/01/2025 20:13, Jeff Adamczak wrote:
We use -maxidle. We use a secondary dependency system outside of dagman to manage a special case where a job releases several other dependent jobs as it runs. The dependent jobs start on hold. We increase maxidle to account for those dependent
jobs. It can get up to around 7k sometimes.
Send HTCondor-users mailing list submissions to
htcondor-users@xxxxxxxxxxx
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
or, via email, send a message with subject or body 'help' to
htcondor-users-request@xxxxxxxxxxx
You can reach the person managing the list at
htcondor-users-owner@xxxxxxxxxxx
When replying, please edit your Subject line so it is more specific
than "Re: Contents of HTCondor-users digest..."
Today's Topics:
1. Community Query of DAGMan Functionality (Cole Bollig)
----------------------------------------------------------------------
Message: 1
Date: Fri, 20 Dec 2024 17:26:11 +0000
From: Cole Bollig <cabollig@xxxxxxxx>
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] Community Query of DAGMan Functionality
Message-ID:
<CO6PR06MB7266D83F44A8FF2114A3671DAF072@xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx>
Content-Type: text/plain; charset="iso-8859-1"
Hi All,
For those of you that interact in DAGMan in some fashion regularly, do you use the -maxidle command line option or the DAGMAN_MAX_JOBS_IDLE configuration option. If so, what was the use case that brought about you setting one of these options?
Cheers,
DAGMan Developer at CHTC
Cole Bollig
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www-auth.cs.wisc.edu/lists/htcondor-users/attachments/20241220/d09a27c5/attachment.html>
------------------------------
Subject: Digest Footer
_______________________________________________
HTCondor-users mailing list
HTCondor-users@xxxxxxxxxxx
The archives can be found at:
https://www-auth.cs.wisc.edu/lists/htcondor-users/
------------------------------
End of HTCondor-users Digest, Vol 133, Issue 43
***********************************************
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
The archives can be found at: https://www-auth.cs.wisc.edu/lists/htcondor-users/
|
![[Computer Systems Lab]](http://www.cs.wisc.edu/pics/csl_logo.gif){kind=logo}