[Condor-users] My dag is frozen

[Lucia Santamaria <lucia.santamaria@xxxxxxxxxx>]

Hi Steffen and condor experts,

for the last 4 days my dagman in morgane seems frozen and doesn't trigger 
more jobs. The dag is not yet finished, as you can see if you execute

lucia@morgane:~/month4_latest/857232370-859651570$ 
/home/lucia/opt/s5_2yr_lowcbc_20080603/lalapps//bin/lalapps_ihope_status 
--dag-file=ihope.dag

(playground is done up to cat4, injection jobs only up to cat1)

however nothing happens

lucia@morgane:~/month4_latest/857232370-859651570$ condor_q lucia
-- Submitter: morgane.merlin2.aei.mpg.de : <10.100.200.91:39685> : 
morgane.merlin2.aei.mpg.de
 ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
70996.0   lucia           7/2  15:20  15+19:13:20 R  0   7.3 
condor_dagman -f -
167274.0   lucia           7/12 23:16   5+15:29:59 R  0   7.3 
condor_dagman -f -

and also, _no_ rescue dag is created. If I look at one of the 
dag.dagman.out corresponding to one of the subdags that are not yet 
finished (for instance, cat2 dags in nsbhinj):

--> in 
nbshinj/inspiral_hipe_nsbhinj_cat2_veto.NSBHINJ_CAT_2_VETO.dag.dagman.out
(tail of the file ... for 4 days now)
7/18 14:05:26   Node 20abc05cccfa0bf1b7e41fa441b90524, Condor ID 169496, 
status STATUS_SUBMITTED
7/18 14:15:26 319886 seconds since last log event
7/18 14:15:26 Pending DAG nodes:
7/18 14:15:26   Node 20abc05cccfa0bf1b7e41fa441b90524, Condor ID 169496, 
status STATUS_SUBMITTED
7/18 14:25:26 320486 seconds since last log event
7/18 14:25:26 Pending DAG nodes:
7/18 14:25:26   Node 20abc05cccfa0bf1b7e41fa441b90524, Condor ID 169496, 
status STATUS_SUBMITTED
7/18 14:35:26 321086 seconds since last log event
7/18 14:35:26 Pending DAG nodes:
7/18 14:35:26   Node 20abc05cccfa0bf1b7e41fa441b90524, Condor ID 169496, 
status STATUS_SUBMITTED

Anybody has a clue what might be happening?
Thanks,
Lucia

--
--------------------------------------------
Lucia Santamaria
Max-Planck-Institut fuer Gravitationsphysik
Albert-Einstein-Institut
Am Muehlenberg 1, 17746 Golm, Germany
Office: +49(0)331-567-7181
---------------------------------------------