[HTCondor-users] Process list of files

["Roger Herikstad" <roger.herikstad@xxxxxxxxx>]

Hi everyone, 
 I recently started testing HTCondor as a replacement for our ageing SGE setup. Our lab uses Mac Pro's as combined compute nodes/workstations, and though SGE works reasonable well with this setup, the fact that it is no longer open source (and has a somewhat convoluted build/install process), made me look around for alternatives. I've managed to get HTCondor up and running with few of our nodes, and so far I really like what I'm seeing. I have one question, though. Our typical SGE workflow would be something like this:

From the terminal:

for s in $( ls <some file pattern> ); do <some setup>; echo "cd $PWD; <some command with arguments> $s" | qsub <sge arguments>; done

In other words, we would have a list of files containing data to be processed, and we would send of one SGE job per file. What is the best way to achieve this workflow under HTCondor? I came across this: 
https://htcondor-wiki.cs.wisc.edu/index.cgi/wiki?p=VaryArgumentsByProcId
which gets me part of the way, but if I understand it correctly, I'd have to manually define the list of files as a ClassAd in the submit file. This is certainly doable, but knowing my users, the more steps you have go through to submit jobs, the less likely they are to use the system. 
i'd appreciate any thoughts on this from the list. Thanks!


Roger Herikstad
Research Fellow
SiNAPSE
National University of Singapore

Attachment: signature.asc
Description: OpenPGP digital signature