Re: [Condor-users] 2 questions about the condor scheduler

[Alain Roy <roy@xxxxxxxxxxx>]

On Nov 6, 2008, at 1:17 PM, Mohamed Salah Meddeb wrote:
> I will work on real time HD video coding; I’ll implement an  
> algorithm of video coding on grid computing because of the  
> computational needed by the algorithms of video coding such as H264  
> (MPEG 4 part 10).I've chosen Condor because it is dedicated for the  
> high throughput computing

Real-time is very different from high-throughput. What exactly do you  
mean by real-time? What are your constraints?

> Please, I have 2 questions:
>   1- Is condor recommended for the multimedia application especially  
> when we work with HD resolution i.e. a tremendous amount of data  
> required for coding

Think of it like this: you write a program to do something--in your  
case HD video coding--and you have a bunch of computer. Condor will  
run your program on a bunch of computers. While your program is  
running, the overhead of running Condor is fairly small. So Condor may  
or may not be recommended. The question is: can the computers you have  
run your program?

>       2- I’ll work on real time HD video coding, after some  
> researches I realized that condor is not suitable for constraint of  
> real time.
> Is it possible to make some modification on condor’s scheduler to  
> make it support the real time constraint?

Define what you need for real-time. How strict is your schedule?

In general, Condor is not optimized for quick job startup, but once  
the job is running Condor adds almost no overhead. The way we ship  
Condor, it may take many seconds, perhaps a minute, to start up a job.  
In some cases, you can speed up the job startup time, but it will  
still be on the order of a few seconds.

That said, if you are writing new code and you are willing to fit into  
an existing framework, you might be interested Condor's Master Worker  
(MW) framework. Instead of jobs, you write code that thinks about  
"tasks". MW runs worker jobs that start up, then grab tasks and run  
them. They can run many tasks, so the overhead of stating up a task is  
much smaller than starting up a job. Depending on what you need for  
real-time, it may be good enough for you, but we don't think of it as  
a system that can meet strict real-time demands, it's just a fairly  
fast system.

http://www.cs.wisc.edu/condor/mw/

-alain

-----------------------------------------------------------------
Alain Roy                                         roy@xxxxxxxxxxx
Condor Project                      http://www.cs.wisc.edu/condor