Re: [HTCondor-users] Using Multiple PRE_SKIP statements in .dag file

[Vinay Kusuma <vinay@xxxxxxxxxxxxx>]

Hello Mark,

"Multiple PRE_SKIP statements with different exit code: this is allowed, as long as you only use one PRE_SKIP statement per node in the dag. If you try adding more, the last one will override anything you've done previously. What you've done here looks fine."

This point was really valuable and helpful.

and I was able to get my DAG running.

So, Previously I was using a single PRE SCRIPT check_input.py and i checked for 3 possible scenarios(exit status 0,1,2).

Now, I created two different PRE SCRIPT, one being check_A.py and other being check_B.py

And, I skip the required nodes using these scripts.

So, now i am doing

SCRIPTÂ ÂPREÂ ÂCREATE_BÂ Âlogs/check_A.py

SCRIPTÂ ÂPREÂ ÂTARGET_BÂ Âlogs/check_A.py

PRE_SKIPÂ ÂCREATE_B 1

PRE_SKIPÂ TARGET_BÂ 1

SCRIPTÂ PREÂ PICK_A logs/check_B.py

SCRIPTÂ PREÂ STAT_A logs/check_B.py

PRE_SKIPÂ PICK_AÂ 1

PRE_SKIPÂ STAT_AÂ 1

Previously, Somehow the node which was supposed to exit with 1 using PRE SCRIPT was exiting with status 2 becauseÂ

all conditions were listed in just on PRE SCRIPT.

Thank you for the help.

Really appreciate it.

Vinay Kusuma,

Junior Bioinformatician,

e: vinay@xxxxxxxxxxxxx

CONFIDENTIALITY NOTICE: This message and any attachments are solely for the use of the intended recipient and may contain privileged, confidential or other legally protected information. If you are not the intended recipient, please destroy all copies without reading or disclosing their contents and notify the sender of the error by reply e-mail.

On Thu, Apr 11, 2019 at 2:59 AM Mark Coatsworth <coatsworth@xxxxxxxxxxx> wrote:

HI Vinay, to answer your questions:

Multiple PRE_SKIP statements with different exit code: this is allowed, as long as you only use one PRE_SKIP statement per node in the dag. If you try adding more, the last one will override anything you've done previously. What you've done here looks fine.

As for ABORT-DAG-ON, the order of this declaration does not matter. If you add one of these rules for a node, and the node fails with the error code you specified, DAGMan will immediately stop its running jobs. So yes, it could be causing the premature exit of other nodes. But it would stop the dag and all its jobs entirely, not just one specific job.

What's the parent-child relationship between all the nodes you mentioned below?

Mark

On Tue, Apr 9, 2019 at 11:07 PM Vinay Kusuma <vinay@xxxxxxxxxxxxx> wrote:

Hello Everyone,Â

Hope you all are doing well. I am Vinay, recently started working on HTC Condor as a part of my project. I want to skip few nodes depending on the input file provided, as of now i have written a pre script which checks my input and sends a exit status and this exit status is captured and used to skip or run a node in PRE_SKIP step.

The scenario is this,Â

For a particular pipeline, I have three possible inputs. Either 'A' file, 'B' file or Both 'A' and 'B' file.

So, When 'B' file is the input, exit status given by pre script is 1

when 'A' files is the input, exit status given by pre script is 2

when both B' and 'A' files are used as input, exit status is 0.

The pre script name is check_input.py

My PRE SCRIPT and PRE_SKIP statements in .dag file are :-

SCRIPTÂ PREÂ Â ÂCREATE_B logs/check_input.py

SCRIPTÂ PREÂ Â ÂTARGET_BÂ Â logs/check_input.py

PRE_SKIPÂ Â Â Â CREATE_B 2

PRE_SKIPÂ Â Â Â TARGET_BÂ Â 2

SCRIPTÂ PREÂ Â ÂPICK_A logs/check_input.py

SCRIPTÂ PREÂ Â ÂSTAT_A logs/check_input.py

PRE_SKIPÂ Â Â Â PICK_A 1

PRE_SKIPÂ Â Â Â STAT_A 1

So, When i am running 'B' file as input, the exit status is 1. The nodes belonging to 'A' files in the above code as skipped as expected by exit status of 1 but when the execution reaches CREATE_B node, this node is exited by same exit status of 1.

So, After analysing the situation, I believe there can be two issues :-

1) Is it allowed to use multiple PRE_SKIP statements with different exit code as it did above?

2) Does the order of ABORT-DAG-ON of nodes have any impact on the premature exit of above node?Â

Looking forward to your reply.

Thank you.

Regards,

Vinay Kusuma

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--

Mark Coatsworth

Systems Programmer

Center for High Throughput Computing

Department of Computer Sciences

University of Wisconsin-Madison

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/