hi...seems like condor should work for this. you will need to have some sort of script/program which creates jobs for condor to handle.
i'm new to the condor list, so be patient!!
i'm trying to get a better understanding of condor, and whether it can be used in a project i'm considering. i'm looking at how to create a distributed platform for creating a web crawling application.
the application will consist of ~1000-2000 perl scripts that are used to parse a given web site. i'd like to have a small network of linux/windows servers be used as client machines that actually run the perl apps, with the management of the client servers/perl apps being handled by some master/parent app/server. i'd like the ability to be able to have a basic round-robin type of approach when determining how/when an app is run on a given machine. (something along the lines of if a client machine is up, and has the available resources (cpu/mem/etc..) then run the next app.)
Condor is just a bunch of command line tools. I've made a WWW based interface which is targeted towards my use of Condor but i assume you can do the same.i'd like to be able to integrate the management/operations of the management/queuing app using a web based interface as well.
i'm curious as to whether condor can easily handle the management taskCondor is not too hard. I've got it working in a mixxed environment of Windows and Linux. File system / permissions will be the largest headahce if you have a mixxed OS situation. you have to have entries for users in samba and windows or just use very relaxed sambe security and you will need to disable condors authentication checking....
without too much complexity, or is condor overkill for what i'm envisioning.