Re: [Condor-users] Third-party file staging with Condor-G

[Jaime Frey <jfrey@xxxxxxxxxxx>]

On Jan 30, 2008, at 10:59 AM, Jan Ploski wrote:

> The following message thread shows (I think) that third-party file  
> staging
> (from/to a GridFTP server different than the submission host) is not
> implemented in Condor-G and currently not possible:
>
> https://lists.cs.wisc.edu/archive/condor-users/2006-July/ 
> msg00059.shtml
>
> The most recent suggestion in that thread was to use a substitution  
> macro
> in order to refer to the matched target GT4 host:
>
> https://lists.cs.wisc.edu/archive/condor-users/2006-July/ 
> msg00128.shtml
>
> However, I can't see how this alone could solve the staging problem.  
> The
> required transferCredentialEndpoint must refer to a piece of data  
> which
> 1) has to be created by Condor before the WS-GRAM job is created (this
> possibly only occurs if you specify input_files/output_files?)
> 2) has to be known by id to the job submitter - but I didn't find any
> macro which would expand to this id
>
> So I guess I have to repeat the original question: is WS-GRAM-level
> third-party file staging possible with Condor-G somehow,  
> nevertheless? If
> not, are there any plans to implement it? The quickest hack which  
> would
> already bring big improvement would be to make it possible to inject  
> one
> or more <transfer> elements into the Condor-generated  
> <transferRequest>
> element.


Third-party file staging is something we're interested in add to  
Condor-G, but there hasn't been enough user demand for us to spend the  
time implementing it.

You can fake it by disabling all Condor-managed file transfer in your  
submit file and adding your own transfer directives like so:

transfer_executable = false
transfer_input = false
transfer_output = false
transfer_error = false
remote_initialdir = <somewhere>
should_transfer_files = NO
globus_xml = <file staging directives>

This requires you to do your own credential delegation for the  
transfers.

Thanks and regards,
Jaime Frey
UW-Madison Condor Team