Re: [Condor-users] Condor commands getting stuck

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

Diana Lousa wrote:
> Hello,
>
> We have made a new installation of condor in our cluster, in the 
> beginning of this week. In this new installation we upgraded from condor 
> 6.8 to version 7.4 and we also changes our dedicated scheduler from a 
> machine with a Fedora OS to one with Ubuntu 10.04.
> We have condor installed in 2 shared directories (one that has binaries 
> for fedora OS and another that has binaries for ubuntu OS) and each 
> machine runs the release correspondent to its OS. Everything ran fine in 
> the first days (from Monday until today), but today the condor commands 
> started getting stuck. Fist condor_q stopped responding and after a few 
> minutes all the jobs just died (without our intervention). We then 
> restarted condor in all our machines, resubmitted the jobs and the same 
> thing happened again after a while (about 15 minutes). Next, we cleaned 
> all our condor log files, killed the deamon in all the machines and 
> restarted the system and submitted a small number of jobs to see how it 
> handled them. Everything was ok for a few hours and now, I'm trying to 
> submit more jobs and the command condor_submit gets stuck. The strangest 
> thing is that the jobs are submitted and start running, but the 
> condor_submit command does not terminate by itself.
> All our system is based on nfs.
>
> Can anyone help?

In your condor_config file(s), is even LOCAL_DIR specified to be a path 
on an NFS mounted volume?  If so, I would try moving LOCAL_DIR (i.e., 
Condor's spool, log, and execute directories) to a local non-NFS file 
system, especially on your submit machine(s) where you are observing 
trouble running condor_submit.

Perhaps as jobs start up, the jobs themselves start pounding hard on the 
NFS server (esp if you have a lot of jobs running), and the submit 
machine's condor_schedd daemon is being blocked trying to access its 
spool subdirectory.

regards
Todd