Re: [Condor-users] Steering Jobs to a File on HDFS

[Matthew Farrellee <matt@xxxxxxxxxx>]

There is no one supported method right now. Condor provides multiple 
mechanisms to achieve your goal, and there are number of people working 
on different approaches.

When the set of data-files is small, some have found advertising 
information about blocks in slot ads is a workable approach. The data 
info is then available to job RANK and REQUIREMENTS expressions. Others 
have looked into extending the ClassAd language with a verb that 
calculates how much of a file is present on a given node, and then used 
that information primarily in RANK expressions (though REQUIERMENTS is 
also possible).

Maybe someone who is actively doing this now can speak up and share some 
of their setup and code.

Best,


matt

On 09/16/2010 06:23 PM, Dvorak DDV wrote:
> Hi Experts,
>
> I see activities of using HDFS as the underlying FS for Condor. As far
> as I understand condor is copying the file out of HDFS before a job
> runs, is this correct?
>
> Since HDFS can be exported to workloads (jobs) directly via e.g. Fuse. I
> am just wondering is it possible that, in a condor cluster where compute
> worker nodes are also HDFS storage nodes, condor can send the job
> directly to the node where the file is located on HDFS?
>
> Thanks.
>
>
> On Wed, Jul 21, 2010 at 4:24 PM, Yushu Yao <yao.yushu@xxxxxxxxx
> <mailto:yao.yushu@xxxxxxxxx>> wrote:
>
>     Hi Developers,
>
>     In the 2009 what't new in Condor talk (
>     http://www.cs.wisc.edu/condor/CondorWeek2009/condor_presentations/tannenba_roadmap_2009.ppt
>     ), page 32, it is said:
>
>     "Thinking about how Condor can steer jobs to the data
>     Via a ClassAd function used in the RANK expression?"
>
>     Just wondering is this done now? If yes is there documentations on
>     how to do it?
>
>     Thanks a lot!
>
>     -Yushu
>     +-------------------------------------------------+
>     | Yushu Yao
>     | Ph:1-510-486-4690
>     |
>     | Lawrence Berkeley National Lab
>     | Mailstop 50B-6222
>     | 1 Cyclotron Road
>     | Berkeley CA 94720-8147 - USA
>     +-------------------------------------------------+
>
>
>
>
>
>
>
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/