Re: [Condor-users] Does stork avoid retransfering data?

[Mag Gam <magawake@xxxxxxxxx>]

Makeflow looks cool!


On Tue, Jan 11, 2011 at 11:55 AM, Douglas Thain <dthain@xxxxxx> wrote:
> Thomas -
>
> You might consider using Makeflow for this task:
> http://www.nd.edu/~ccl/software/makeflow
>
> The idea is that you express your tasks in Makeflow, submit a bunch of
> 'worker' processes to Condor, and Makeflow will distribute tasks among
> the workers.  If they have some common executables and input files,
> they will be automatically cached at the workers, so you don't have to
> keep transmitting them.
>
> Cheers,
> Doug
>
> On Tue, Jan 11, 2011 at 8:27 AM, Rowe, Thomas <rowet@xxxxxxxxxx> wrote:
>> I have to run a simulation about a thousand times with different seeds.  The
>> simulation executable and data total about 100MB.  This sounds like a job
>> for DAGMan & Stork, because this 100MB collection of files needs to get
>> copied around reliably, and some large output files need to be transferred
>> back to the originating machine reliably.
>>
>>
>>
>> My question: Does Stork and/or DAGMan do anything intelligent about avoiding
>> recopying files?  The input files are identical for all thousand runs; only
>> the seed varies.  But I would like to have Condor manage each run
>> individually.  So does all the data and the executable get copied around a
>> thousand times, cleaned up after each run?  If the thousand reps are child
>> to the Stork job that transfers files in place, does everything just work
>> with no extraneous recopying of input data?
>>
>>
>>
>> Thanks,
>>
>> Thomas Rowe
>>
>> _______________________________________________
>> Condor-users mailing list
>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/condor-users/
>>
>>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>