Re: [Condor-users] condor-g data transfer mechanism

["Maryam Khademi" <m.khademi@xxxxxxxxx>]

Hi Ewa,


Yeah, I am pretty familiar with Pegasus. Had even long question&answer with your developers (Garang and the rest) before.

You are right, it is of course better to transfer data just in case of need back to submitter. But my question was about condor-g mechanism itself and how it works!

One of the guys mentioned in answer to this post, that using the transfer_input_files, and transfer_output_files parameter causes submitter to act as an intermediary. I am wondering using the pure Condor-G (i.e., not to add stage-in/out tasks to DAG), would it possible to transfer intermediate data between grid-sites without submitter involvement?! If so, then which kind of parameters should I use in my submit files?

Many Thanks,

Maryam

On Sat, Dec 6, 2008 at 7:05 AM, Ewa Deelman <deelman@xxxxxxx> wrote:

Hi Maryam,

 

I am not sure what you mean by Condor-G meta-scheduling.

In our work we use the Pegasus Workflow Management System (http://pegasus.isi.edu) to map an abstract (resource-independent) DAG onto the distributed resources. As part of the mapping, Pegasus adds nodes to the DAG to stage the data to the execution sites and move the results off. 

Our workflow execution engine, DAGMan uses Condor-G to send the jobs in the workflow to the grid sites. The results do not have to come back to the submitter. If you don't want to save the intermediate products, they will just exist on the remote sites for the duration of the workflow and only the final data will be brought back to where you want it to go.  If intermediate data needs to be moved between sites, it will be done via 3^rd party, so that the submitter is not involved.

 

For large data is it usually better to just transfer it from A to B directly.  When jobs are mapped to the same site and there is shared file system on that site, B can access the data directly.

 

Please let me know if you have any questions,

 

Thanks,

-Ewa

 

 

From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Maryam Khademi
Sent: Thursday, December 04, 2008 7:06 PM
To: Condor-Users Mail List
Subject: [Condor-users] condor-g data transfer mechanism

 

Hi,

I am using Condor-G to execute a DAG. As I understand its mechanism, the meta-scheduler dispatches the jobs among different grid sites. Meanwhile, all the intermediate data are transfered to the submitter machine, so that if a job on grid site A needs the output of another job on grid site B, it can have it through submitter. Is that correct? Following this method, how about dependent jobs mapped to the same grid sites?! Is the input data again being asked from submitter?

Thanks,
Maryam

_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/