Re: [Condor-users] Windows: Any ideas on how to Transfer-in direc tory?

[Matt Hope <matthew.hope@xxxxxxxxx>]

Sorry didn't spot your original post.

 

We use tar (and bzip2/gzip) to handle the structure successfully (though a dodgy version of tar created files which condor could not access thus leaving turd directories all over the farm.

 

The file transfer mechanism does nothing with directories and I know no way round it that doesn't involve changing the state per completion.

 

An alternate if you don't want to rely on some tool set is to flatten your directory structure with renaming to indicate directory structure (essentially replacing / with the char of you choice and escaping any existing instances of it). This is very unpleasant compared to TAR though...
 

Matt
 

On 8/30/05, Froebe, Scott <froebes@xxxxxxx> wrote:

Well, since no-one seemed to have any other suggestions, I guess using TAR would have to be the solution then.

 

-----Original Message-----
From: Froebe, Scott [mailto: froebes@xxxxxxx]
Sent: August 5, 2005 2:16 PM
To: Condor-users (E-mail)
Subject: [Condor-users] Windows: Any ideas on how to Transfer-in directory?

Hi All,

I just hit upon a snag for one of the programs we run.  One method of using our program requires subdirectories (of input) to the PWD of it's execution.  Since the standard file transfer mechanism in Condor for windows makes a flat directory, what can I do to preserve the structure?  I've had ideas like using tar/zip/etc, but our resulting output may be several gigabytes, so this would be slow and may result with a file bigger than the file system could handle.

As a little background, we have 2 clusters, a development one and a production one.  The development cluster is running the Bristol stuff so we map file shares (required due to a specific developer user requirement).  For the production cluster I am trying to stay away from the added complexity and network overhead of the Bristol stuff.  As such this issue is only relevant to our production cluster.

Thanks for any ideas.