[Condor-users] Using Dropbox has a Condor pool job submission mechanism

[Ian Cottam <ian.cottam@xxxxxxxxxxxxxxxx>]

Has anyone thought about, or tried, using Dropbox <www.getdropbox.com> 
as a job submission mechanism to a Condor pool?

What I have in mind is the sort of pool that comprises some large number 
of nodes on a private network, with one node -- the submit node -- 
having two network cards: the second being on the global network (i.e. 
the visible Internet). Every user has an account on the submit node, and 
they have to ssh in (or similar) and copy files over; submit the job; 
monitor for it finishing; copy results out.

(If the submit node is also on a shared network file system, also 
mounted on users' machines, the below is invalidated, but I think there 
are many instances where that is not the case.)

The scenario is that all the users use 2GB free Dropbox accounts. (It 
would still work if, like me, they already had [paid] accounts). The 
single job submit node 'owns' a paid account of 50 or 100GB.
When one becomes a user of the pool, a folder is shared, via Dropbox, 
from the job submit node account with the user's account. Users do not 
need login accounts on the submit node.

Then, to submit a Condor job into the private pool, a user would simply 
move or copy the set of files required into the shared folder. A simple 
monitor program on the job submit node would need to look for jobs 
arriving and condor_submit them. The results simply appear back on the 
user's remote machine courtesy of the DropBox sync mechanism.

Clearly, there would be some subtleties of implementation; for example: 
a Condor job submit file would have to have a standardised name; and one 
would need a mechanism for ensuring DropBox had synchronised all the 
files before the job was run; and no doubt others.

Just a thought, but I would be interested in reading people's comments.
regards
-Ian







--
Ian Cottam
Information Systems Manager
Manchester Interdisciplinary Biocentre &
School of Materials &
School of Chemical Engineering and Analytical Science (Room C24)
The John Garside Building (Room G.002)
The University of Manchester
e: ian.cottam@xxxxxxxxxxxxxxxx
t: 0161 306 5198
m: 07856 849831
http://personalpages.manchester.ac.uk/staff/Ian.Cottam
[Giving a talk in MIB? Upload it: http://ic.mib.man.ac.uk/mibtheatre.php]