Re: [HTCondor-users] using hdfs with HTCondor awaring data locality

[Dan Bradley <dan@xxxxxxxxxxxx>]

I suspect the gains to be had are smaller than you may expect.  However, an example of a classad function that does this may be found here:

https://github.com/scooperstein/hdfs_best_hosts_caplugin

Instead of directly embedding calls to this function in the job rank _expression_, we introduced a layer of indirection.  The job rank _expression_ refers to an attribute that is updated periodically by condor_job_router, which evaluates the custom classad function to determine the list of most desireable machines for the job to run on.  This way, the custom function only needs to be linked into condor_job_router, and the rest of condor is never slowed down by performing this query.

One problem with this idea is what happens when the whole cluster is busy.  When a machine finishes running a job, the scheduling problem is not which machine the next job would prefer but which job would be best to run on the idle machine.  The condor_schedd is optimized for the former type of scheduling.  We have worked on a small patch to the schedd to make it do the other type of scheduling when desired.  This is not yet in a release of condor.

--Dan

On 10/24/13 4:02 PM, Tim St Clair wrote:

Easy... no.

I've pondered many options, but the most viable are:

1.) Create a classad function which determines distance(file_block, machine) and use that in a rank _expression_.  This is not pretty, because you would need to do that per block.

2.) Create a YARN based application. 

... 

Cheers,

Tim

---

From: "ricky lee" <rickylee0815@xxxxxxxxx>
To: htcondor-users@xxxxxxxxxxx
Sent: Thursday, October 24, 2013 3:45:09 PM
Subject: [HTCondor-users] using hdfs with HTCondor awaring data locality

Hi all,

I saw a link of using HDFS with HTCondor. 

http://research.cs.wisc.edu/htcondor/manual/v7.7/8_2Using_Condor.html

After deploying HDFS through condor, I want to submit a condor job with a requirement of HDFS data locality aware. For example, a job requirement should look like

requriements=startd_has_specific_HDFS_blcok

Is there an easy way to express the HDFS data locality with Condor? thx.  

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/

--

Cheers,
Tim

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/