[Condor-users] problem with matched job sitting idle

[Armen Babikyan <armenb@xxxxxxxxxx>]

Hi,

In my Condor setup, I'm using Condor's DAG functionality to ensure order 
of dependencies between different programs, and to maximize efficiency 
of the pipeline through my system.  I'm having a problem though:

In two consecutive stages of the pipeline (called B and C, 
respectively), interaction with some external hardware device occurs.  
The design of the hardware allows for pipelined control - the iteration 
#N of stage B can occur while iteration #(N-1)'s stage C is taking 
place.  Furthermore, there is only one of these hardware resources 
available.  I've written a proxy between Condor and this hardware, and 
have two programs which (should) effectively run and exit right after 
each other.

My experiment generates lots of these "B -> C" DAGs.  Among several job 
B's that will appear as idle in 'condor_q', one will get to run, and the 
other job B's will sit idle, waiting for the first job B to exit.  When 
the job B exits, Condor will schedule a job C, and another  job B.

When only one B -> C process is occuring, the system runs fine: B runs, 
B exits, condor schedules C, C runs, C exits, pipeline continues.

The problem I'm having is that with more than one "B -> C" DAG, the 
second B-job runs, but the first C-job sits idle in 'condor_q' forever.  
I'm not sure why.  The single machine controlling this external hardware 
has two VM's on it, (configured with NUM_CPUS = 2. It also has 
hyperthreading, but that shouldn't matter, i'm pretty sure).  I've made 
sure to define my two resources (in the machine and job configurations), 
and add one resource to each of the VM1_STARTD_EXPRS and 
VM2_STARTD_EXPRS variables in the machine's config.  IOW, Job B and Job 
C require different resources (e.g. JOB_B and JOB_C, the former provided 
by VM1 and the latter by VM2).

I looked up C's job number among the log files of the machines in my 
cluster, and none have any mention of the job.  I can only find mention 
of my job in the spool directory of the submitting machine.  'condor_q 
-analyze' has this to say about the C job that won't run:

129.000:  Run analysis summary.  Of 50 machines,
    49 are rejected by your job's requirements
     0 reject your job because of their own requirements
     0 match but are serving users with a better priority in the pool
     0 match but reject the job for unknown reasons
     0 match but will not currently preempt their existing job
     1 are available to run your job

Any ideas or advice would be most helpful.  Thanks!

 - Armen

--
Armen Babikyan
MIT Lincoln Laboratory
armenb@xxxxxxxxxx . 781-981-1796