Re: [Condor-devel] too many links in spool directory

[Dan Bradley <dan@xxxxxxxxxxxx>]

On 10/12/10 11:58 AM, Matthew Farrellee wrote:
> On 10/12/2010 12:47 PM, Dan Bradley wrote:
> > Hi all,
> >
> > If you care about the spool directory, please take a glance at the
> > following ticket which proposes adding a hierarchy of directories.
> >
> > I'd like to finish working on it this week (in time for the CHEP
> > conference next week!), so that we can demonstrate that Condor scales to
> > more than 32k running jobs.
> >
> > https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=1706
> >
> > Thanks,
> > --Dan
>
> Not spitting hairs - "Condor scales to more than 32K running jobs" -> 
> "a single Condor Schedd scales to moer than 32K running jobs"?
>
> willb has done work to avoid spool creation for jobs that do not need 
> them. I understand your workload may require the spool, but if not you 
> may use his work to test >32K without code changes.
>
> Any idea why we haven't run into this problem much in the past? It is 
> not uncommon to have >32K spooled jobs.

In the tests we are doing, we do not use spooling, but we do use file 
transfer.  The code that exists (tip of the master branch) still creates 
the job spool directories.

I believe that if we were running condor as root, the spool directories 
would not get created, because the schedd has been optimized not to 
create them but the shadow still tries to create them.  The shadow does 
this with USER_PRIV and would therefore fail when running as root but 
succeed when running in personal condor mode.

--Dan