HTCondor Project List Archives



[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-devel] too many links in spool directory




On 10/12/10 11:58 AM, Matthew Farrellee wrote:
On 10/12/2010 12:47 PM, Dan Bradley wrote:
Hi all,

If you care about the spool directory, please take a glance at the
following ticket which proposes adding a hierarchy of directories.

I'd like to finish working on it this week (in time for the CHEP
conference next week!), so that we can demonstrate that Condor scales to
more than 32k running jobs.

https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=1706

Thanks,
--Dan
Not spitting hairs - "Condor scales to more than 32K running jobs" -> 
"a single Condor Schedd scales to moer than 32K running jobs"?
willb has done work to avoid spool creation for jobs that do not need 
them. I understand your workload may require the spool, but if not you 
may use his work to test >32K without code changes.
Any idea why we haven't run into this problem much in the past? It is 
not uncommon to have >32K spooled jobs.
In the tests we are doing, we do not use spooling, but we do use file 
transfer.  The code that exists (tip of the master branch) still creates 
the job spool directories.
I believe that if we were running condor as root, the spool directories 
would not get created, because the schedd has been optimized not to 
create them but the shadow still tries to create them.  The shadow does 
this with USER_PRIV and would therefore fail when running as root but 
succeed when running in personal condor mode.
--Dan