Re: [Condor-devel] Condor as a vm scheduler

[Matt Hope <Matt.Hope@xxxxxxxxxxxxxxx>]

-----Original Message-----
From: Matthew Farrellee [mailto:matt@xxxxxxxxxx] 
Sent: 14 January 2010 14:13
To: Matt Hope
Cc: condor-devel@xxxxxxxxxxx
Subject: Re: [Condor-devel] Condor as a vm scheduler
 
> > Not really (though it sounds sensible as a general rule) it's the
> > *intra* negotiation cycle issues rather than inter negotiation
> > cycles.

> Do you often go from 0 to 100 in your workflows? Is it more HPC like?
Our farm is deliberately provisioned such that it has more capacity than raw throughput would require for three main reasons.

 * The latency of an individual's task when it decomposes to a significant (several hundred or thousand individual jobs) is (significantly) reduced.
 * When large numbers of users are running jobs everyone gets significant throughput
  * people tend to work on things at the same time in office hours so there is considerable overlap
 * Certain classes of jobs require far more from a machine that just the cpu

That final one is the kicker, when those jobs run the machine in question has plenty of cycles to spare (multi core era) but is often disk limited (disk size in blades is an ongoing issue) so the maximum throughput for a certain set of jobs is infact determined by the machine axis, not the core (slot) axis.

Our jobs neatly fall into categories of the monsters above (single threaded) or low memory (well <3GB), low disk but really (likewise single threaded) CPU hungry.
A further wrinkle is that the farm is 'split' down the slot axis (as opposed to machine axis) with per slot RANK and START expressions.

As such it is quite often the case that the farm will go from a low utilization to high very quickly. 

It's still firmly HTPC, with tasks tending to have hours or days worth of time but latency to get started (and catch easy to find errors) is often frustrating to the users so this aspect is something that we would rather measure in seconds rather than minutes.

Matt


----
Gloucester Research Limited believes the information provided herein is reliable. While every care has been taken to ensure accuracy, the information is furnished to the recipients with no warranty as to the completeness and accuracy of its contents and on condition that any errors or omissions shall not be made the basis for any claim, demand or cause for action.
The information in this email is intended only for the named recipient.  If you are not the intended recipient please notify us immediately and do not copy, distribute or take action based on this e-mail.
All messages sent to and from this email address will be logged by Gloucester Research Ltd and are subject to archival storage, monitoring, review and disclosure.
Gloucester Research Limited, 5th Floor, Whittington House, 19-30 Alfred Place, London WC1E 7EA.
Gloucester Research Limited is a company registered in England and Wales with company number 04267560.
----