Re: [Condor-users] condor_rm cannot remove very large number of jobs?

[Dan Bradley <dan@xxxxxxxxxxxx>]

The activity that is typically responsible for the schedd taking a long 
time to remove jobs is a lot of small disk transactions.  Some 
optimizations have been done in the 7.5 series to improve this (about a 
30% speedup in my tests of job removal).

If you put user logs and the schedd spool on a disk with a faster 
response time (e.g. an SSD), you should see noticeably better performance.

--Dan

On 12/4/10 3:31 PM, Matthew Farrellee wrote:
> That is often the problem...
>
> http://spinningmatt.wordpress.com/2009/12/16/timeouts-from-condor_rm-and-condor_submit/ 
>
>
> Best,
>
>
> matt
>
> On 12/03/2010 07:50 PM, Steven Timm wrote:
> > There have long been issues with huge numbers of jobs exiting at
> > once in condor. they keep on improving performance but
> > I don't think they are are up to 20K yet.
> > Look at the various timeouts including the tool timeouts--I think
> > if you lengthen a timeout you can probably get a condor_rm of
> > 20K jobs to finish. IF that doesn't work then up the debug level
> > in the schedd log and that will tell the story.
> >
> > Steve
> >
> >
> > On Fri, 3 Dec 2010, Rob wrote:
> >
> > > Hi,
> > >
> > > I encountered this when 'stress-testing' my condor pool.
> > >
> > > Occassionally I submit 20.000 jobs as a single cluster to the queue,
> > > which is
> > > OK.
> > > However, removing jobs with condor_rm does not work when there are so
> > > many jobs
> > > in the queue to be removed.
> > >
> > > For example:
> > > ----------------------------------
> > > $ condor_q | grep " 350." | wc -l
> > > 14515
> > >
> > > $ condor_rm 350
> > > Couldn't find/remove all jobs in cluster 350.
> > >
> > > $ condor_rm -all
> > > Could not remove all jobs.
> > > ----------------------------
> > >
> > > I then use a script:
> > >
> > > #!/bin/sh
> > > counter=0
> > > while [ $counter -lt 20000 ]
> > > do
> > > condor_rm 350.${counter} >> /dev/null 2>&1
> > > counter=$(( $counter + 1))
> > > done
> > > exit
> > >
> > > which removes the jobs one-by-one and this works.
> > > (though this script takes a long time!)
> > >
> > >
> > > What is the problem when removing all jobs at once with condor_rm ?
> > >
> > > When the number of jobs to be removed is less (say about 4000), then
> > > condor_rm
> > > can remove all at once!
> > >
> > >
> > > I am using condor 7.4.2 on Fedora Linux.
> > >
> > > Regards,
> > > Rob.
> > >
> > >
> > >
> > > _______________________________________________
> > > Condor-users mailing list
> > > To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx 
> > > with a
> > > subject: Unsubscribe
> > > You can also unsubscribe by visiting
> > > https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> > >
> > > The archives can be found at:
> > > https://lists.cs.wisc.edu/archive/condor-users/
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/