Re: [HTCondor-users] using hdfs with HTCondor awaring data locality

[Tim St Clair <tstclair@xxxxxxxxxx>]

Easy... no.

I've pondered many options, but the most viable are:

1.) Create a classad function which determines distance(file_block, machine) and use that in a rank _expression_.  This is not pretty, because you would need to do that per block.

2.) Create a YARN based application. 

... 

Cheers,

Tim

---

From: "ricky lee" <rickylee0815@xxxxxxxxx>
To: htcondor-users@xxxxxxxxxxx
Sent: Thursday, October 24, 2013 3:45:09 PM
Subject: [HTCondor-users] using hdfs with HTCondor awaring data locality

Hi all,

I saw a link of using HDFS with HTCondor. 

http://research.cs.wisc.edu/htcondor/manual/v7.7/8_2Using_Condor.html

After deploying HDFS through condor, I want to submit a condor job with a requirement of HDFS data locality aware. For example, a job requirement should look like

requriements=startd_has_specific_HDFS_blcok

Is there an easy way to express the HDFS data locality with Condor? thx.  

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--

Cheers,
Tim