Re: [condor-users] running globus jobs without a shared filesystem

[Dan Bradley <dan@xxxxxxxxxxxx>]

mcal00@xxxxxxxxxxxxx wrote:

> There's a line in the message you mention that says:
>
> "This isn't much of a problem for output files, as Condor can be told transfer
> all output files from the job (those created or modified after the job began
> running)."
>
> Well, I could never get it to do it.

I believe the confusion here is whether you are talking about stage-back 
of files from the execution node to the gatekeeper node (which is what 
Alain was talking about), or whether you are talking about stage-back of 
files to the original point of submission, which is what I think you are 
talking about, right Mark?

In any case, there is _no_ option in Condor-G to automatically stage 
back output files to the submission site.  I am right now updating the 
manual in several places where the different behavior for Globus 
universe is not made clear.  The transfer-files options Alain was 
talking about only handle copying back files from the execution machine 
to the gatekeeper machine.

An additional transfer must usually take place between the gatekeeper 
and the submit machine.  If you know the files in advance, you can 
specify them explicitly in the transfer_output_files list.  This 
mechanism (new in Condor 6.5) uses GRAM file staging.  I personally do 
not know if GRAM file staging is robust enough to handle large data flows.

Probably the most common technique is to submit a follow-up "fork" job 
that sends files back via gsiftp.  It sounds like you have a solution 
similar to this which effectively spawns off a fork job automatically.  
You may want to investigate how well this scales, if you are likely to 
be submitting large numbers of jobs, because the gatekeeper tends to be 
quite heavily burdened.

Dan Bradley
University of Wisconsin, Condor Project


Condor Support Information:
http://www.cs.wisc.edu/condor/condor-support/
To Unsubscribe, send mail to majordomo@xxxxxxxxxxx with
unsubscribe condor-users <your_email_address>