Re: [Condor-users] Using Dropbox has a Condor pool job submission mechanism

[Ian Cottam <ian.cottam@xxxxxxxxxxxxxxxx>]

Thanks very much for that Ian.
I don't know Andy in Physics but I do know many colleagues in Research 
Computing Services here in Manchester. In fact, I just sent them the 
'thought variation' below for comment:
===
Hi ...,
 I've had a thought about how we could make it simpler for HTC/HPC 
users to submit jobs to the local cloud of resources we have now and 
that will doubtless grow considerably.

It involves using Dropbox <www.getdropbox.com>. I expect you are 
familiar with it, but, if not, it is a great cross-platform file sync 
(+backup) system. Anyone can get a 2GB account free and you can rent 
50/100GB accounts for a small amount (I have a 50GB one for my 4 Macs 
for example).

My 'thought experiment' really needs a diagram, but I'm useless with 
such (especially within emails!).

Each Condor Pool, Grid, or HPC is usually front-ended with a node that 
is also on the global network (so people can ssh in, or similar). I 
propose that these nodes have paid 50/100GB Dropbox accounts. When a 
user requests access to a facility, the relevant node shares a free 2GB 
folder with him/her. (One user could have several such shares across the 
range of facilities on campus.)

When the user want to run, say, a Condor job he/she simply moves or 
copies the set of files necessary to the shared folder. Dropbox would 
then automatically kick in and make sure it was synchronised with the 
corresponding folder on the head node. All we would have to write is a 
simple script that looks for such jobs arriving and submit them to the 
job queue.

When the job finishes and because the output files go to the shared 
folder they will be copied by DropBox to the user's PC.

Some consequences:
1. Users need not install systems, such as Condor, to use them.
2. The interface is trivial and uniform across all the facilities (job 
submission files may differ of course).
3. There is no need to have user accounts on the resources (YES!).
4. There is no need to login/ssh/sftp/etc to the resource (which may be 
running a platform foreign to the user).
5. If the job fails, the user only gets the log back; there is no 
interaction with the job.
6. The script to monitor and then submit the job needs to know all the 
files have arrived. This just needs some agreed protocol, but no doubt 
there would be (other) subtleties of implementation.
===

regards
-Ian

Ian Stokes-Rees wrote:
> Hi Ian,
>
> I have spent quite a bit of time thinking of something very similar to 
> this.  You are using the file system as the message queue.  It is quite 
> a reasonable approach, in my opinion.
>
> I'm not sure it is necessary to do anything which is "dropbox" specific 
> -- dropbox may just add on additional features (e.g. auto-syncing shared 
> files back to user machines).
>
> You could easily imagine clients that are simply configured with account 
> access details and a path, and all the rest is based on configuration 
> files and job files found in particular directories.
>
> If you have someone at Manchester who is interested in putting together 
> a proof of concept around this, I'd be very happy to contribute.
>
> Do you know Andrew McNab in the Physics department?  He is, in my 
> opinion, quite a visionary thinker when it comes to HPC/grid computing, 
> and has been "in the game" for a long time.  If you can get a bit of his 
> time, he may be able to give you some good ideas/comments.
>
> Ian
>
> Ian Cottam wrote:
> > Has anyone thought about, or tried, using Dropbox <www.getdropbox.com> 
> > as a job submission mechanism to a Condor pool?
> ...
> >   
>
> --
> Ian Stokes-Rees                            W: http://sbgrid.org
> ijstokes@xxxxxxxxxxxxxxxxxxx               T: +1 617 432-5608 x75
> SBGrid, Harvard Medical School             F: +1 617 432-5600
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at: 
> https://lists.cs.wisc.edu/archive/condor-users/

--
Ian Cottam
Information Systems Manager
Manchester Interdisciplinary Biocentre &
School of Materials &
School of Chemical Engineering and Analytical Science (Room C24)
The John Garside Building (Room G.002)
The University of Manchester
e: ian.cottam@xxxxxxxxxxxxxxxx
t: 0161 306 5198
m: 07856 849831
http://personalpages.manchester.ac.uk/staff/Ian.Cottam
[Giving a talk in MIB? Upload it: http://ic.mib.man.ac.uk/mibtheatre.php]