Re: [HTCondor-users] Transfer out files question

["Dan, Bowen" <bowen.dan@xxxxxxxx>]

Hi Mick,

 

Thanks for your quick response.  I have tried the following:

 

TRANSFER_OUTPUT_FILES = A/myfile.txt, B/myfile.txt

TRANSFER_OUTPUT_REMAPS = âA/myfile.txt=a_file.txt;B/myfile.txt=b_file.txtâ

 

However, it looks to me that the myfile.txt was transferred back to the submit machine first, then the remap logic applied to it. Because of the file name collision, condor only transferred back one myfile.txt, so TRANSFER_OUTPUT_REMAPS = âA/myfile.txt=a_file.txt;B/myfile.txt=b_file.txtâ did not work.

 

Due to some reasons, I do not have control over how files are created or named on the execute machine. Can condor rename the file before transferring it back? Or can condor only transfer the directory with just selective files under it?

 

Thanks a lot!

 

Best,

Bowen

 

 

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of "Mick T." <timony@xxxxxxxxxxxxxxxxxxx>
Reply-To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Date: Wednesday, August 30, 2017 at 11:54 AM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Transfer out files question

 

I think you need to remove the backslashes, and replace

 

TRANSFER_OUTPUT_FILES = A/myfile.txt/, B/myfile.txt/ 

 

with 

 

TRANSFER_OUTPUT_FILES = A/myfile.txt, B/myfile.txt

 

Tho' that may put the files in the current directory and overwrite one of the files with the other. So you may want to use the transfer_output_remaps option, or easier still rename both files so there's name collision.

 

See the docs at: http://research.cs.wisc.edu/htcondor/manual/current/condor_submit.html

 

"When a path to an output file or directory is specified, it specifies the path to the file on the execute side. As a destination on the submit side, the file is placed in the job's initial working directory, and it is named using the base name of the original path. For example, path/to/output_file becomes output_file in the job's initial working directory. The name and path of the file that is written on the submit side may be modified by using transfer_output_remaps. Note that this remap function only works with files but not with directories."

 

Cheers!

-- 

Mick Timony
Team Lead & Scientific Web Services Developer
SBGrid Core - SBGrid.org
Harvard Medical School
--

 

On Wed, Aug 30, 2017 at 12:20 PM, Dan, Bowen <bowen.dan@xxxxxxxx> wrote:

Hi,

 

I have a question regards to how the transfer out files work.  Currently on the execute machine, I have two folders A and B, both contains a considerable amount of data, the two directories both have a file named âmyfile.txtâ. It would be like this:

A/myfile.txt and B/myfile.txt. Instead of transferring whole directories (A and B) back to the submit machine, I only want to transfer A/myfile.txt, B/myfile.txt. So far, I have tried the following:

 

TRANSFER_OUTPUT_FILES = A/myfile.txt/, B/myfile.txt/

 

Unfortunately, it did not work. I was wondering if there is a way for me to only transfer the desired files back?

 

Appreciate your help!

 

Best,

Bowen

 

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/