Re: [HTCondor-users] Condor submit for multiple small jobs

[Brian Candler <B.Candler@xxxxxxxxx>]

On Thu, May 16, 2013 at 05:59:43PM +0530, Batsayan Das wrote:
>    I am trying to understand the common practice used to submit multiple
>    small jobs with low overhead. I have a pool of small jobs. I do not
>    want  to invoke condor_submit each and every time for each job.

I have found that dagman works well for this, as described at
http://research.cs.wisc.edu/htcondor/manual/current/2_10DAGMan_Applications.html#SECTION0031013000000000000000

By default, dagman only submits 5 jobs every 5 seconds, but you can crank
this up by setting e.g.

  DAGMAN_MAX_SUBMITS_PER_INTERVAL = 200

in your condor_config.local

Useful tools:

  condor_submit_dag -maxidle 500 foo.dag
  (to stop your queue getting too huge at one time)

  condor_q -dag
  (shows DAG node names)

  condor_q -run -dag
  (shows DAG node names and which host each job is running on)

More info:
http://research.cs.wisc.edu/htcondor/manual/current/2_10DAGMan_Applications.html
http://research.cs.wisc.edu/htcondor/manual/current/condor_submit_dag

If the jobs are almost identical, you can submit a "cluster" of jobs in a
single submission:

  executable = myjob.sh
  arguments = $(process)
  queue 1000

This will run myjob.sh 0, myjob.sh 1 ... myjob.sh 999

More info:
http://research.cs.wisc.edu/htcondor/manual/current/condor_submit.html

Other options you could consider are the python API and the SOAP API for
submitting jobs, but I have not tried these.

HTH,

Brian.