Re: [HTCondor-users] Condor + Docker?

[Greg Thain <gthain@xxxxxxxxxxx>]

On 07/10/2014 01:57 PM, Branden Timm wrote:
> This may be a better question for htcondor-devel, but I wonder if 
> anybody has thought of or tried using Condor in conjunction with 
> Docker?  I can imagine several advantages that could be leveraged via 
> Docker, including -


> 1) Deploying jobs to containers running pre-configured images 
> containing all the software for that job (instead of, say, having 
> software pre-installed across your cluster or setting up/tearing down 
> requirements on job start/completion).  Docker images tend to be 
> lightweight, and they're cached once they're downloaded, so on 
> repeated runs almost nothing would have to be transferred to the 
> execute node.
> 2) Taking advantage of the underlying cgroups features of the Linux 
> kernel to actually limit jobs to the requirements specified in their 
> submit file while the system is under pressure (cpu, memory, and disk 
> IO can all be throttled).  Of course, the other advantage is that it's 
> a share-based system, so you can guarantee a minimum amount of CPU 
> access while still allowing it to burst beyond that on an otherwise 
> idle system.
> 3) Complete namespace isolation (process, filesystem, and network) of 
> running jobs from the rest of the system.


Docker is a technology that us HTCondor folks are keeping an eye on.  
Note that for bullet point #2 above, HTCondor today has support for 
using cgroups to sandbox the cpu and memory usage of a job, and to 
measure resource usage.  HTCondor also optionally support process 
isolation on Linux systems with kernel support.

-Greg