Re: [HTCondor-users] Job being queued when script echoes nothing

[Marco van Zwetselaar <zwets@xxxxxxxxxx>]

    Thanks John, this is on latest (10.2.0), with direct submit.
    
    I'd argue that a node in a DAG should fail if and only if one or
    more of its submitted jobs fail. If there are no jobs, then none
    fail.
    
    The practical application here is rerunning a DAG or batch job where
    part of the output is already present. A local script listing the
    remaining work items can then be used in "queue X, Y from
    ./list-todo.sh |".
    
    More generally: if a job is intended to process a variable number of
    work items (using "queue ... from"), then it's conceivable that
    there are 0 items to process. Why treat this as an exception?
    
    Whether DAGman should consider this a success or failure is a
    separate issue though. What currently happens is that DAGman
    (contrary to condor_submit, which doesn't produce a job), actually
    submits the job (once), with empty values for the "queue ..."
    parameters.
    
    Cheers,
    Marco

On 28/02/2023 02:28, John M Knoeller via HTCondor-users wrote:

A DAG is not going to know how to deal with a submit file that does not submit any jobs â should it treat that as a failure? A success?

Â

You do not say what version of HTCondor, or of DAG man. Is this a modern versions that does direct submit? Or an older version that is using condor_submit to submit the jobs?

Â

-tj

Â

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Marco van Zwetselaar
Sent: Sunday, February 26, 2023 2:21 PM
To: John M Knoeller via HTCondor-users <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Job being queued when script echoes nothing

Â

Hi John,

This old issue bit me again, so I double checked. What's happening is that the bug occurs only when the job is in a dag.

With the following:

 echo-nothing.sh:
ÂÂÂ #!/bin/sh
ÂÂÂ exit 0

 empty.job:
ÂÂÂ executable = /usr/bin/echo
ÂÂÂ arguments = "[$(ARG)]"
ÂÂÂ output = job.out
ÂÂÂ queue ARG from ./echo-nothing.sh |

 empty.dag:
ÂÂÂ JOB empty empty.job

Submitting empty.job correctly gives the warning and queues no jobs, whereas "condor_submit_dag empty.dag" surprisingly queues and executes the job.

Cheers
Marco

On 22/09/2022 18:09, John M Knoeller via HTCondor-users wrote:

I think your script must be outputting something because when I try with a script that prints nothing I see this

Â

Submitting job(s)

WARNING: "fromcmd.sub" has only empty "queue" commands -- no jobs queued

Â

Or perhaps there is some complicating factor in your submit file. Could you send me a submit file and script that reproduces the problem?

Â

-tj

Â

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Marco van Zwetselaar
Sent: Wednesday, September 21, 2022 6:04 PM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Job being queued when script echoes nothing

Â

Dear all,

I regularly use this construct in my job files:

ÂÂÂ queue ID, FN from ./list-todo.sh |

e.g. to skip jobs that have already been done.

However when all jobs have been done, and ./list-todo.sh echoes nothing (not even a newline), Condor still queues a job. The ID and FN are set to '', so the job usually fails, whereas IMO it shouldn't have been queued in the first place.

An obvious workaround is to add a wrapper script, but that isn't very clean: the "non-job" still gets scheduled and executed, consuming resources it won't use. Worse, if $(FN) appears in transfer_input_files, then this won't work.

I would think this is a bug, but maybe there's a knob somewhere that changes this behaviour?

Thanks,
Marco

--

Marco van Zwetselaar

Bioinformatician

Kilimanjaro Clinical Research Institute

P.O. Box 2236 | Moshi, Kilimanjaro | Tanzania

http://www.kcri.ac.tz/ÂÂ|ÂÂzwets@xxxxxxxxxxÂÂ|ÂÂ+255 782 33412

_______________________________________________

HTCondor-users mailing list

To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a

subject: Unsubscribe

You can also unsubscribe by visiting

https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

Â

The archives can be found at:

https://lists.cs.wisc.edu/archive/htcondor-users/

Â

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/