Re: [HTCondor-users] Job being queued when script echoes nothing

[John M Knoeller <johnkn@xxxxxxxxxxx>]

A DAG is not going to know how to deal with a submit file that does not submit any jobs â should it treat that as a failure? A success?

 

You do not say what version of HTCondor, or of DAG man.  Is this a modern versions that does direct submit? Or an older version that is using condor_submit to submit the jobs?

 

-tj

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Marco van Zwetselaar
Sent: Sunday, February 26, 2023 2:21 PM
To: John M Knoeller via HTCondor-users <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Job being queued when script echoes nothing

 

Hi John,

This old issue bit me again, so I double checked. What's happening is that the bug occurs only when the job is in a dag.

With the following:

  echo-nothing.sh:
    #!/bin/sh
    exit 0

  empty.job:
    executable = /usr/bin/echo
    arguments = "[$(ARG)]"
    output = job.out
    queue ARG from ./echo-nothing.sh |

  empty.dag:
    JOB empty empty.job

Submitting empty.job correctly gives the warning and queues no jobs, whereas "condor_submit_dag empty.dag" surprisingly queues and executes the job.

Cheers
Marco

On 22/09/2022 18:09, John M Knoeller via HTCondor-users wrote:

I think your script must be outputting something because when I try with a script that prints nothing I see this

 

Submitting job(s)

WARNING: "fromcmd.sub" has only empty "queue" commands -- no jobs queued

 

Or perhaps there is some complicating factor in your submit file.  Could you send me a submit file and script that reproduces the problem?

 

-tj

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Marco van Zwetselaar
Sent: Wednesday, September 21, 2022 6:04 PM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Job being queued when script echoes nothing

 

Dear all,

I regularly use this construct in my job files:

    queue ID, FN from ./list-todo.sh |

e.g. to skip jobs that have already been done.

However when all jobs have been done, and ./list-todo.sh echoes nothing (not even a newline), Condor still queues a job. The ID and FN are set to '', so the job usually fails, whereas IMO it shouldn't have been queued in the first place.

An obvious workaround is to add a wrapper script, but that isn't very clean: the "non-job" still gets scheduled and executed, consuming resources it won't use. Worse, if $(FN) appears in transfer_input_files, then this won't work.

I would think this is a bug, but maybe there's a knob somewhere that changes this behaviour?

Thanks,
Marco

--

Marco van Zwetselaar

Bioinformatician

Kilimanjaro Clinical Research Institute

P.O. Box 2236 | Moshi, Kilimanjaro | Tanzania

http://www.kcri.ac.tz/  |  zwets@xxxxxxxxxx  |  +255 782 33412

_______________________________________________

HTCondor-users mailing list

To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a

subject: Unsubscribe

You can also unsubscribe by visiting

https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:

https://lists.cs.wisc.edu/archive/htcondor-users/