<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8"> </head> <body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""> To the best of my knowledge, SLURM has no facilities for transferring job files between machines. It assumes you have a shared filesystem for all job files. That’s why you don’t see any directives in slurm_submit.sh. <div class="">The BLAHP doesn’t copy job files from a local filesystem to a shared one on the submit machine. It should probably give an error if it detects that job files are on a local filesystem and the batch system can’t move them, but that currently doesn’t happen.</div> <div class=""><br class=""> </div> <div class="">For your current testing, all of the job files (including the original job script) should be on the shared filesystem. In your ultimate setup, the HTCondor spool directory will need to be on the shared filesystem on your custom Scarf node. Also, submission from the other HTCondor node will have to include spooling of job files (either Condor-C or condor_submit -remote).</div> <div class=""><br class=""> </div> <div class=""> <div class=""> <div class=""> <div class=""> - Jaime<br class=""> <div><br class=""> <blockquote type="cite" class=""> <div class="">On Jul 30, 2018, at 11:20 AM, Brian Ritchie - UKRI STFC <<a href="mailto:brian.ritchie@stfc.ac.uk" class="">brian.ritchie@stfc.ac.uk</a>> wrote:</div> <br class="Apple-interchange-newline"> <div class=""> <div class="WordSection1" style="page: WordSection1; caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> I'm trying to use HTCondor to submit jobs to our Scarf HPC. At<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> present, this uses Platform LSF, and (following initial work by Andrew<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Lahiff) I've managed to get this to work (to some extent). However,<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Scarf is replacing Platform LSF with Slurm, and I'm having trouble<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> getting submission to work with Slurm in the case where the jobscript<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> is in a directory that is not shared with the worker nodes. (I am<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> submitting from a custom Scarf node that has Condor<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> installed. Ultimately, jobs will be submitted to this node from an<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> HTCondor node that is external to Scarf, so sharing won't be an<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> option.)<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""> </o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> The problem seems to be that the jobscript that is generated by BLAH's<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> slurm_submit.sh assumes that the original jobscript has been copied to<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> a (unique) filename in a sandbox folder, but the copy never happens.<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> The lsf_submit.sh script generates BSUB directives that (I think)<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> instruct LSF to perform the intial copy, but I see no equivalent in<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> slurm_submit.sh.<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""> </o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> None of this is reflected in the files created by HTCondor: the log<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> file implies that the job ran OK (but consumed no resources), and the<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> output and error files are always empty. Only by modifying the blah<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> scripts to log to somewhere other than /dev/null (and copying the<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> generated jobscripts to file) was I able to get more information about<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> what was going wrong!<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""> </o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> batch_gahp.config has many options for defining which directories are<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> shared, and for overriding default locations for sandboxes etc. I have<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> tried numerous permutations, to no avail.<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""> </o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Is there a better guide to configuration than the comments in batch_gahp.config?<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> What special considerations are required for Slurm?<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""> </o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Thanks,<o:p class=""></o:p></div> <div style="margin: 0cm 0cm 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> Brian</div> </div> </div> </blockquote> </div> </div> </div> </div> </div> </body> </html>