Re: [HTCondor-users] Job being queued when script echoes nothing

Mailing List Archives Authenticated access	UW Madison Computer Sciences Department Computer Systems Lab

Thanks John, this is on latest (10.2.0), with direct submit.

I'd argue that a node in a DAG should fail if and only if one or more of its submitted jobs fail. If there are no jobs, then none fail.

The practical application here is rerunning a DAG or batch job where part of the output is already present. A local script listing the remaining work items can then be used in "queue X, Y from ./list-todo.sh |".

More generally: if a job is intended to process a variable number of work items (using "queue ... from"), then it's conceivable that there are 0 items to process. Why treat this as an exception?

Whether DAGman should consider this a success or failure is a separate issue though. What currently happens is that DAGman (contrary to condor_submit, which doesn't produce a job), actually submits the job (once), with empty values for the "queue ..." parameters.

Cheers,
Marco

On 28/02/2023 02:28, John M Knoeller via HTCondor-users wrote:

A DAG is not going to know how to deal with a submit file that does not submit any jobs – should it treat that as a failure? A success?

You do not say what version of HTCondor, or of DAG man. Is this a modern versions that does direct submit? Or an older version that is using condor_submit to submit the jobs?

-tj

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Marco van Zwetselaar
Sent: Sunday, February 26, 2023 2:21 PM
To: John M Knoeller via HTCondor-users <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Job being queued when script echoes nothing

Hi John,

This old issue bit me again, so I double checked. What's happening is that the bug occurs only when the job is in a dag.

With the following:

echo-nothing.sh:
    #!/bin/sh
    exit 0

empty.job:
    executable = /usr/bin/echo
    arguments = "[$(ARG)]"
    output = job.out
    queue ARG from ./echo-nothing.sh |

empty.dag:
    JOB empty empty.job

Submitting empty.job correctly gives the warning and queues no jobs, whereas "condor_submit_dag empty.dag" surprisingly queues and executes the job.

Cheers
Marco

On 22/09/2022 18:09, John M Knoeller via HTCondor-users wrote:
I think your script must be outputting something because when I try with a script that prints nothing I see this

Submitting job(s)

WARNING: "fromcmd.sub" has only empty "queue" commands -- no jobs queued

Or perhaps there is some complicating factor in your submit file. Could you send me a submit file and script that reproduces the problem?

-tj

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> On Behalf Of Marco van Zwetselaar
Sent: Wednesday, September 21, 2022 6:04 PM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Job being queued when script echoes nothing

Dear all,

I regularly use this construct in my job files:

queue ID, FN from ./list-todo.sh |

e.g. to skip jobs that have already been done.

However when all jobs have been done, and ./list-todo.sh echoes nothing (not even a newline), Condor still queues a job. The ID and FN are set to '', so the job usually fails, whereas IMO it shouldn't have been queued in the first place.

An obvious workaround is to add a wrapper script, but that isn't very clean: the "non-job" still gets scheduled and executed, consuming resources it won't use. Worse, if $(FN) appears in transfer_input_files, then this won't work.

I would think this is a bug, but maybe there's a knob somewhere that changes this behaviour?

Thanks,
Marco

--

Marco van Zwetselaar

Bioinformatician

Kilimanjaro Clinical Research Institute

P.O. Box 2236 | Moshi, Kilimanjaro | Tanzania

http://www.kcri.ac.tz/ | zwets@xxxxxxxxxx | +255 782 33412
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
 
The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

Mailing List Archives

Authenticated access

Re: [HTCondor-users] Job being queued when script echoes nothing