Re: [Condor-users] unsure about potential application

[Matthew Farrellee <matt@xxxxxxxxxx>]

On 08/15/2010 03:46 PM, andrew.downs@xxxxxxxx wrote:
> Greetings, I'm trying to find out if Condor is the right software for
> what I'm doing. My application (q3map2.exe) is a type of compiler that
> assembles map data for Quake 3 Arena. The kinds of things I've designed
> for it have become so complex that I think a small cluster might be in
> order. While reading through Condor documentation, I saw this:
>
> "Multi-process jobs are not allowed. This includes system calls such as
> fork(), exec(), and system()."

That info is dated at best, and currently only remotely true in the 
context of Condor's Standard Universe. Avoiding fork/exec/system and 
open sockets is a requirement if you link with Condor's user-space 
checkpointing library and expect checkpointing to work. Probably not 
something you're looking for atm anyway.

Condor will happily run any program that runs unattended, i.e. in some 
sort of batch mode.


> q3map2 can be multi-threaded for either SMP or multi-core
> configurations. Is that the same as being multi-process?
>
> My long term plan was to have 4 systems, each with a 6-core AMD chip all
> chewing through the same data set (but different parts) at the same
> time, in other words, my aim is to have q3map2 utilize the computational
> resources of each machine as if all of the cores in the separate systems
> were all in one system.
>
> My vague understanding of Condor is that if you submit a job (presumably
> q3map2.exe) Condor would find all of the available machines and split up
> the work between them in some fashion.

That would be pretty incredible. Condor isn't going to give your program 
a unified view of the systems, as if they were one, or break up your 
problem space automatically. Once you've broken up your problem space 
into independent pieces, what Condor will do is all the scheduling of 
processing, management of input/output files, process execution 
management, failed process re-execution, provide progress and 
utilization information, etc.


> What leads me to believe this is:
>
> Myth: Condor requires users to recompile their applications.
> *Reality: Condor runs ordinary, unmodified applications.*
> *
> *
> *I'm not even sure how that's possible, but if it is, it's incredible
> and perfect for what i'd like to do.*
> *
> *
> Is condor too much for what I'm trying to do? Should I be looking
> somewhere else? Reading something else?
>
> Thanks for your time.

"Condor runs ordinary, unmodified applications" - totally true. If you 
do recompile you can get additional features. The information in the 
manual is from an era when people thought checkpointing and Condor's 
Standard Universe (maybe because of the name?) were the only way Condor 
would run jobs.

Condor certainly isn't too much for what you're trying to do. It can be 
a pretty lightweight system. It contains a lot of features you'll 
blissfully ignore at first and maybe get interested in later on.

Best,


matt