[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [HTCondor-users] Access to renamed DAG node names
- Date: Mon, 12 Jan 2015 22:25:29 +0000
- From: Brian Candler <b.candler@xxxxxxxxx>
- Subject: Re: [HTCondor-users] Access to renamed DAG node names
On 12/01/2015 15:42, R. Kent Wenger wrote:
Thank you. I missed $(JOB) in "DAGMan Applications" - probably because
this special macro can only be expanded in a VARS statement.
Yes, this is pretty easy to do:
In your DAG file:
Job NodeA A.sub
Vars NodeA nodename=$(JOB)
output = $(nodename).out
error = $(nodename).err
$(nodename) will expand to the modified node name.
Incidentally, quotes are needed:
VARS NodeA nodename="$(JOB)"
> you're saying that for failed nodes you'd like to see the values of
output and error put directly into the JOBSTATE.LOG or NODE_STATUS
files, so you could find them without having to look at the submit file?
Not quite; I'm saying that if a particular node fails, I want to be able
to read and display its associated output and error files (without
having to parse the DAG file to locate the submit file, then parse the
submit file, then expand all the nested macros and VARS)
I hadn't thought of putting output/error into the JOBSTATE.LOG or
NODE_STATUS files; I am just planning to stick to a convention that I
use <nodename>.out and <nodename>.err as the names of the files, so that
they are trivial to find given the nodename.
My difficulty was in finding a way to set the input and output
attributes in the submit file to the right nodename, in the presence of
dagman's node name munging - and you've given me the answer to that now.
P.S. For completeness, here's the final script for two DAGs, with all
files in the top-level directory.
---- 8< ----
cat <<EOS >"$2.dag"
JOB A foo.sub
VARS A job="A" input="$1" arg1="aaa" arg2="AAA" nodename="\$(JOB)"
JOB B foo.sub
VARS B job="B" input="$2+A.out" arg1="bbb" arg2="BBB" nodename="\$(JOB)"
SCRIPT POST B deliver.sh \$JOB
cat <<EOS >foo.sub
universe = vanilla
executable = foo.sh
arguments = "'\$(arg1)' '\$(arg2)'"
output = \$(nodename).out
error = \$(nodename).err
cat <<'EOS' >foo.sh
chmod +x foo.sh
cat <<'EOS' >deliver.sh
cat "$1.out" >>/tmp/deliver.out 2>>/tmp/deliver.err
chmod +x deliver.sh
echo "hello world" >data.in
write_dag data.in dag1
write_dag dag1+B.out dag2
cat <<EOS >dag.dag
SPLICE dag1 dag1.dag
SPLICE dag2 dag2.dag
PARENT dag1 CHILD dag2
# condor_submit_dag dag.dag
# Final output is in dag2+B.out
---- 8< ----