[Condor-users] Condor based Meta Data Collector/Negotator for cross domain pooling

[Adam Lathers <alathers@xxxxxxxxxxxxxx>]

Dear condor gurus,

    I'm new to the condor world as it is, but I've got a rather  
sizable task.  I've seen some stuff on the mailing list that looks  
similar to what I need, but not enough to give me a clue on how to  
address it.

    Basically, I have several sets of resources.  I have stand alone  
work stations, that are organized as one pool, with a dedicated  
collector/negotiator.  I have a cluster, that's on a private network,  
with one front-end, that's doing IP Masquerading, organized as a  
pool.  The collector is listening on the public interface, and all  
the nodes report to that interface.  On top of all of this, I also  
have external resources.  to the best of my knowledge, we have no  
administrative domain with these resources, but they do accept our  
signing policy for x509 proxy certs, and they are all running both  
Condor and Globus.

    SO......my challenge.  I need to basically pool these resources  
all together.  The "ideal view" would basically be to have one meta  
collector/negotiator, for lack of a better term.  A single machine  
that can give stats, and do match making, even abstractly to the  
various pools and such. From the sound of things, there are bits and  
pieces of what I need around, but I haven't figured out how to get  
all the parts together.  For example, I can pole a pool with  
something like, condor_status -pool collector-name.domain.edu.  And I  
can set up flocking from this meta host to the various pools, but  
that doesn't solve my out of domain issue.  I'd REALLY like to have a  
single, replicate-able, solution, so even if the local resources have  
a little more overhead and config work, it's fine, cause I'd rather  
support one variation on this configuration, rather than having a  
specific config for each pool that this meta host talks to.  Thanx  
all for any advice, reading material, or others.  Again, I have no  
shame admitting I'm a complete newbie, so really really small words,  
and simple questions won't be offensive.

So the Q/A part.

A.) Do I need to do anything further for the cluster pool, as long as  
IP Masquerading is working?  I don't need to do explicit port  
forwarding or anything, right?

B.) Can I make a meta collector, that receives all of the ClassAds  
from the various pool collectors?  Or should I make something that  
actively collects from each pool?  This second option might  
accommodate the "external" pool issue best? (example condor_status - 
long -pool external.domain.edu)

C.) Assuming B, Can I leverage this meta collector to do some form of  
meta negotiation that's homogeneously managed across all these  
resources, even the external pool?

D.) Is this something that can be manageable from the central meta  
negotiator?  Can jobs be started and stopped and such from the meta  
host, or will it not be feasible, since there are various schedd's  
running on the various distributed pools?


_______________________________________________________
Adam Lathers
NCMIR: National Center for Microscopy and Imaging Research
Network Systems Engineer
phone: (858) 822-0735
fax:   (858) 822-0828
web:   http://ncmir.ucsd.edu