Re: [Condor-users] Condor with Ec2

[Matthew Farrellee <matt@xxxxxxxxxx>]

That's a good overview. We use the condor_job_router, which now provides 
some of its own hook call-outs, and its routing rules. The hooks use 
some of Amazon's Web Services (e.g. SQS & S3) to pass the job 
information to a newly created EC2 job, which is the result of the routing.

Best,


matt

William Henry wrote:
> Hi Jon,
>
> This is what we've been doing here at Red Hat.  We have an external agent called the jobrouter and the magic is in its routing capability. We implemented new hooks (i.e. job hooks) to assist in doing this. These new hooks in the job router are similar to the hooks available in the startd and starter.  It also takes advantage of AWS Web services to transfer data in and out.
>
> I've not been the one working on this directly. I've just been doing some testing and playing with it. So I expect someone like Matt will chime in on the topic.
>
> William Henry
> Red Hat, MRG (http://www.redhat.com/mrg)
> Colorado, USA.  +1 719  302 2302
>
> ----- "Jonathan D. Proulx" <jon@xxxxxxxxxxxxx> wrote:
>
> > Hi William,
> >
> > I for one am interested in more details of your aproach.
> >
> > Getting our Condor cluster to dynamicly expand into EC2 space has
> > been
> > in my queue for a while, and may finally be getting back to the top.
> >
> > My thoughts such as they've been is to have an external agent that
> > watches the queue and determines (through magic?) when new AMIs
> > should
> > be spawned ( i'm thinking shutdown logic should probably live on the
> > AMI so it's tied to the billing cycle ), with the AMI mostly a
> > generic
> > resource like our other cluster nodes (using GCB for communication).
> >
> > -Jon
> >
> >
> > On Wed, Dec 03, 2008 at 07:41:47PM -0500, William Henry wrote:
> > :
> > :FYI,
> > :
> > :I posted a couple of articles about using Condor with EC2:
> > :
> > :First just using Condor to start an AMI:
> > :http://ipbabble.com/2008/11/mrg-grid-and-ec2.html
> > :
> > :Second using Condor and some new hooks to schedule jobs in an AMI in
> > EC2:
> > :http://ipbabble.com/2008/12/mrg-and-ec2-enhanced.html
> > :
> > :Not all the details are here. There are some settings needed for the
> > Enhanced (2nd option).
> > :
> > :Let me know if any of you would like those or if you think I should
> > post a more exhaustive example on the blog.
> > :
> > :
> > :Best,
> > :William Henry
> > :Red Hat, MRG (http://www.redhat.com/mrg)
> > :_______________________________________________
> > :Condor-users mailing list
> > :To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
> > with a
> > :subject: Unsubscribe
> > :You can also unsubscribe by visiting
> > :https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> > :
> > :The archives can be found at: 
> > :https://lists.cs.wisc.edu/archive/condor-users/
> > _______________________________________________
> > Condor-users mailing list
> > To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx
> > with a
> > subject: Unsubscribe
> > You can also unsubscribe by visiting
> > https://lists.cs.wisc.edu/mailman/listinfo/condor-users
> >
> > The archives can be found at: 
> > https://lists.cs.wisc.edu/archive/condor-users/
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at: 
> https://lists.cs.wisc.edu/archive/condor-users/