Re: [HTCondor-users] sep/delim/escape for queue vars in file

[John M Knoeller <johnkn@xxxxxxxxxxx>]

you can use $STRING()  to evaluate an arbitrary classad _expression_ as part 

of $() expansion.   it's a bit ugly, but something like this should work.  

  outfiles_expr = replaceall(";", "$(outfiles)", ",")

  transfer_output_files = $STRING(outfiles_expr)

  queue dirname,exename,outfiles,infiles from /workflows/job_f.txt

where job_f.txt has lines like this (spaces added to clarify, they are not needed):

  abc, abc.sh, o1.txt;o2.txt, f1.txt,f2.txt,f3.txt

The other way to do this would be to use the ASCII unit separator character

to separate items in your job_f.txt file.   This is the character just before space

in the ASCII table.  decimal value 31 or hex value 1F.    When the submit reader for the itemdata file sees a unit separator character in a file,  it ignores spaces and commas and splits items only where the unit separators are.    This is how python submit is able to allow a dictionary for the itemdata. 

Using the unit separator character, lines in your job_f.txt file would look like this

   abc<US>abc.sh<US>01.txt,02.txt<US>f1.txt,f2.txt,f3.txt

where <US> above represents the unit separator character.  

-tj

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Klint Gore <kgore4@xxxxxxxxxx>

Sent: Tuesday, November 7, 2023 8:09 PM

To: htcondor-users@xxxxxxxxxxx <htcondor-users@xxxxxxxxxxx>
Subject: [HTCondor-users] sep/delim/escape for queue vars in file

 

I'm trying to do something like this

initialdir=$(dirname)

executable=$(exename)

transfer_input_files=$(infiles)

queue dirname,exename,infiles from /workflows/job_f.txt

where job_f.txt is along the lines of

abc,abc.sh,f1.txt,f2.txt,f3.txt

mmm,mmm.sh,f1.txt,f2.txt

xyz,xyz.sh,f2.txt

Which seems to do what I expect with everything after the 2^nd comma being bundled into the infiles variable.

Then I came to transfer_output_files where I need a 2^nd list and how to deliniate them properly.  transfer_input_files is comma separated, transfer_output_files is comma separated, the job file is comma separated. 

Is there a way to do this?

e.g. if abc.sh produces about 50 output files and I only want 2 (abc.converg, abc.final), what do I put into the abc line to acheive this? assuming that subsequent lines are different thus needing their own values.

delimiter?

abc,abc.sh,?f1.txt,f2.txt,f3.txt?,?abc.converg,abc.final?

separator? eg space

abc,abc.sh,f1.txt f2.txt f3.txt,abc.converg abc.final

escape?

abc,abc.sh,f1.txt\,f2.txt\,f3.txt,abc.converg\,abc.final

klint.