<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> <style type="text/css" style="display:none;"></style> </head> <body dir="ltr"> <div id="divtagdefaultwrapper" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;"> <p>So, an overview:</p> <p><br> </p> <p>I have 3 machines in a condor cluster: herc0, herc1, and starscream. All of them mount home directories from a fourth, optimus. On each of the condor machines I have the home directories mounted at /nfs/optimus/home/ . herc0 has a secondary drive mounted locally as /local_data0 and also for all machines at /nfs/data_disks/herc0b .</p> <p><br> </p> <p>Using the tutorial program, simple.c, I can successfully run the jobs in my home directory. All cores are used, all save write to disk. If I cd into a directory in <span> /nfs/data_disks/herc0b I get the following errors submitting:</span></p> <p><span><br> </span></p> <p><span></p> <div><span style="font-family: "Courier New", monospace;">[zdhughes@herc0 zdhughes]$ condor_submit submit </span></div> <div><span style="font-family: "Courier New", monospace;">Submitting job(s)..............................</span></div> <div><span style="font-family: "Courier New", monospace;">30 job(s) submitted to cluster 81.</span></div> <div><br> </div> <div><span style="font-family: "Courier New", monospace;">WARNING: File /nfs/data_disks/herc0b/users/zdhughes/simple.error is not writable by condor.</span></div> <div><br> </div> <div><span style="font-family: "Courier New", monospace;">WARNING: File /nfs/data_disks/herc0b/users/zdhughes/simple.out is not writable by condor.</span></div> <br> </span> <p></p> <p><span>And the ShadowLog has:</span></p> <p><span><br> </span></p> <p><span><br> </span></p> <p><span></p> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ******************************************************</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** condor_shadow (CONDOR_SHADOW) STARTING UP</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** /usr/sbin/condor_shadow</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** SubsystemInfo: name=SHADOW type=SHADOW(6) class=DAEMON(1)</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** Configuration: subsystem:SHADOW local:<NONE> class:DAEMON</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** $CondorVersion: 8.4.7 Jun 03 2016 BuildID: 369249 $</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** $CondorPlatform: x86_64_RedHat7 $</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** PID = 30540</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** Log last touched 9/9 18:47:43</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ******************************************************</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 Using config source: /etc/condor/condor_config</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 Using local config sources: </span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 /etc/condor/condor_config.local</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 config Macros = 71, Sorted = 71, StringBytes = 1828, TablesBytes = 1176</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ******************************************************</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 CLASSAD_CACHING is OFF</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** condor_shadow (CONDOR_SHADOW) STARTING UP</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** /usr/sbin/condor_shadow</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 Daemon Log is logging: D_ALWAYS D_ERROR</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** SubsystemInfo: name=SHADOW type=SHADOW(6) class=DAEMON(1)</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** Configuration: subsystem:SHADOW local:<NONE> class:DAEMON</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** $CondorVersion: 8.4.7 Jun 03 2016 BuildID: 369249 $</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** $CondorPlatform: x86_64_RedHat7 $</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** PID = 30541</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ** Log last touched 9/9 18:50:42</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ******************************************************</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 Using config source: /etc/condor/condor_config</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 Using local config sources: </span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 /etc/condor/condor_config.local</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 config Macros = 71, Sorted = 71, StringBytes = 1828, TablesBytes = 1176</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 CLASSAD_CACHING is OFF</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 Daemon Log is logging: D_ALWAYS D_ERROR</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 Daemoncore: Listening at <0.0.0.0:19411> on TCP (ReliSock).</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 Daemoncore: Listening at <0.0.0.0:38254> on TCP (ReliSock).</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 DaemonCore: command socket at <10.0.7.10:19411?addrs=10.0.7.10-19411&noUDP></span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 DaemonCore: command socket at <10.0.7.10:38254?addrs=10.0.7.10-38254&noUDP></span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 DaemonCore: private command socket at <10.0.7.10:19411?addrs=10.0.7.10-19411></span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 DaemonCore: private command socket at <10.0.7.10:38254?addrs=10.0.7.10-38254></span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 Initializing a VANILLA shadow for job 81.1</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 Initializing a VANILLA shadow for job 81.0</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 (81.1) (30541): WriteUserLog::initialize: safe_open_wrapper("/nfs/data_disks/herc0b/users/zdhughes/simple.log") failed - errno 13 (Permission denied)</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 (81.1) (30541): WriteUserLog::initialize: failed to open file /nfs/data_disks/herc0b/users/zdhughes/simple.log</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 (81.1) (30541): Failed to initialize user log to /nfs/data_disks/herc0b/users/zdhughes/simple.log</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 (81.1) (30541): Job 81.1 going into Hold state (code 22,0): Failed to initialize user log to /nfs/data_disks/herc0b/users/zdhughes/simple.log</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 (81.1) (30541): RemoteResource::killStarter(): DCStartd object NULL!</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 (81.0) (30540): WriteUserLog::initialize: safe_open_wrapper("/nfs/data_disks/herc0b/users/zdhughes/simple.log") failed - errno 13 (Permission denied)</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 (81.0) (30540): WriteUserLog::initialize: failed to open file /nfs/data_disks/herc0b/users/zdhughes/simple.log</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 (81.0) (30540): Failed to initialize user log to /nfs/data_disks/herc0b/users/zdhughes/simple.log</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 (81.0) (30540): Job 81.0 going into Hold state (code 22,0): Failed to initialize user log to /nfs/data_disks/herc0b/users/zdhughes/simple.log</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 (81.0) (30540): RemoteResource::killStarter(): DCStartd object NULL!</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 18:50:42 ******************************************************</span></div> <br> </span> <p></p> <p><span>etc. for all 30 instances of the job. I have an entry for my the disk in my export file: </span></p> <p><span><br> </span></p> <p><span><span><span style="font-family: "Courier New", monospace;">/local_data0 herc*.lexas(rw) </span><span style="font-family: "Courier New", monospace;">starscream.lexas(rw)</span></span><br> </span></p> <p><span><span><br> </span></span></p> <p><span><span>and I can read/write to the disk as a user. I have chmod 777 the directory zdhughes, which is were the program is located at and the files written to. Going deeper into the directory structure so that many parent directories also have full rwx access does nothing. Additionally, herc0 has a local account, labuser. When executing the (vanilla) job from its home directory the jobs on herc0 run normally (the jobs on other machines hold, as expected); but if I run the job locally from <span style="font-family: Calibri, Arial, Helvetica, sans-serif, "Apple Color Emoji", "Segoe UI Emoji", NotoColorEmoji, "Segoe UI Symbol", "Android Emoji", EmojiSymbols; font-size: 16px;">/local_data0/users/labuser/ and get the same thing:</span></span></span></p> <p><span><span><br> </span></span></p> <p><span><span></p> <div><span style="font-family: "Courier New", monospace;">09/09/16 19:06:20 (82.0) (31628): WriteUserLog::initialize: safe_open_wrapper("/local_data0/users/labuser/simple.log") failed - errno 13 (Permission denied)</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 19:06:20 (82.0) (31628): WriteUserLog::initialize: failed to open file /local_data0/users/labuser/simple.log</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 19:06:20 (82.0) (31628): Failed to initialize user log to /local_data0/users/labuser/simple.log</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 19:06:20 (82.0) (31628): Job 82.0 going into Hold state (code 22,0): Failed to initialize user log to /local_data0/users/labuser/simple.log</span></div> <div><span style="font-family: "Courier New", monospace;">09/09/16 19:06:20 (82.0) (31628): RemoteResource::killStarter(): DCStartd object NULL!</span></div> <div><br> </div> <div>Any ides?</div> <br> </span></span> <p></p> <p><span><span><br> </span></span></p> <p><span><span>Thanks,</span></span></p> <p><span><span><br> </span></span></p> <p><span><span>Zach</span></span></p> </div> </body> </html>