Re: [HTCondor-users] Windows Samba and Condor

[Dennis Zheleznyak <dennis@xxxxxxxxxxxxx>]

Hi Greg ! 

Thank you very much :) 

I ended up adding these lines (marked with ***) to the batch scripts that creates the condor.submit files (small script I wrote):

FOR /F "tokens=* delims=*(*" %%A in (conf_files\database) do (call :submit_file %%A)

:submit_file

SET "last=%~n1"

***echo net use \\servername\sharename\%1 /user:DOMAIN\USERNAME PASSWORD
***echo mkdir %last%                    
***echo copy \\servername\sharename\%1\*.* %last%\

Database file holds a lot of paths that the matlab complied executable needs, then it takes the path and holds it in a variable %%A, and the last part is %last% which strips

the path and takes only the last part of it, for example C:\Programs\Medical\One\8759, it takes only 8759. 

These lines prevent the submit machine from copying the data from the storage server to the "compute" nodes, rather the compute nodes will do that :) 

Thanks again ! 

Dennis.

On Mon, Jan 6, 2014 at 8:48 AM, <Greg.Hitchen@xxxxxxxx> wrote:

You can make a small DOS batch file to use as your Condor “executable”.

 

This batch file would copy across any files it needs, run the “real” executable, copy any results back,

and clean up after itself.

 

This then removes the burden of file transfers from the submit node to each of the individual execute nodes.

 

Cheers

 

Greg

 

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Dennis Zheleznyak
Sent: Monday, 6 January 2014 2:37 PM
To: HTCondor-Users Mail List
Subject: [HTCondor-users] Windows Samba and Condor

 

Hi everyone, 

 

I have a small cluster that consists of 5 Windows machines. 

 

The data it self sits on the domain controller of the environment, all the machines have a network mapped drive named L:\.

 

When submitting a lot of jobs, the submit machine copies the data from the network mapped drive to the "work" nodes - which is a huge drawback, the load on the network is huge and jobs can't start running since most of them are waiting to be copied. 

 

Is there any way to make the "work" nodes copy the data them selves instead of giving the submit to do that ? 

 

Thank you, 

Dennis.

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/