[HTCondor-devel] Future of PrivSep, interested in feedback/opinions

[Todd Tannenbaum <tannenba@xxxxxxxxxxx>]

The list of things that do not work properly if you are running your 
execute nodes with PrivSep enabled keeps growing.  Off the top of my 
head, I boldly claim that condor_tail, job x509 proxy updates (gt #104), 
upcoming Lark work, and most of the job container functionality incl cpu 
affinity and cgroup containers (limits on memory, process tracking, pid 
namespace, file namespace) do not work.  Do folks agree ?  Esp re the 
job container stuff, I am not positive, but I think it is likely borked 
w/ PrivSep.
We have three options: (1) make everything work with PrivSep, (2) 
document all the stuff that stops working if you enable PrivSep and let 
admins do the risk/benefit analysis themselves, or (3) get rid of PrivSep.
Option #1 : I have a feeling for what it would take to fix proxy updates 
(couple days) and condor_tail (few days), but no good feeling re the job 
container work.  At first blush it seems like a really big task (many 
weeks? a few months when all said and done?).  Not sure it is worth months.
Option #2 : Seems like another example of 'punt to the user'.  I think 
most admins would opt for the job container stuff over priv sep.
Option #3 : If we got rid of PrivSep, it would lessen many code paths to 
continue to support, test.  Plus, would anyone miss it?  Is anyone 
beyond UW-Madison even using PrivSep (just sent that question out to 
condor-users)?  My guess is until it is "on by default", very few places 
will ever use it, and again I think most folks would rather see the 
container stuff on by default over privsep. Long term, the container 
stuff may be available to non-root (in RHEL 8 or so), which makes the 
motivation for PrivSep in general less relevant - HTCondor could run as 
the same user as all the jobs, and containers would prevent jobs for 
tamperings w/ the HTCondor daemons or other jobs.
Thoughts? Comments? I am mis-understanding something?

Thanks
Todd

--
Todd Tannenbaum <tannenba@xxxxxxxxxxx> University of Wisconsin-Madison
Center for High Throughput Computing   Department of Computer Sciences
HTCondor Technical Lead                1210 W. Dayton St. Rm #4257
Phone: (608) 263-7132                  Madison, WI 53706-1685