On 03/10/2011 07:17 PM, Brian Bockelman wrote:
On Mar 10, 2011, at 6:06 PM, Matthew Farrellee wrote:On 03/10/2011 06:10 PM, Brian Bockelman wrote:Hi all, Last night, I took another detour into "cool things in modern Linux kernels" and came up with per-job PID namespaces for Condor: https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=1959 Basically, when a job runs, the starter requests the a new PID namespace from the kernel. The clone'd process believes it is PID 1, with all process in the job hanging off that. It looks something like this: [bbockelm@rcf-bockelman condor]$ condor_run ps faux USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND bbockelm 1 0.0 0.0 106200 1132 ? SN 14:15 0:00 /bin/bash /home/bbockelm/projects/condor/.condor_run.17238 bbockelm 2 0.0 0.0 108052 1000 ? RN 14:15 0:00 ps faux However, to the "outside world", these appear as normal processes. The processes inside the job can't view or contact external processes - two jobs running within the same Unix account can't discover or send signals to each other. Additionally, when "PID 1" dies, the kernel wipes out the remaining processes started by the job. It's a fairly neat trick. This all requires kernel 2.6.24 or later. Enjoy! BrianNice. Does the accounting for memory, cpu, io all get rolled up into one process for Condor to monitor too? Bye bye proc family tree?<- expensive to maintainActually, it would be possible to make the proc family tree less expensive by putting the condor_master into its own PID namespace - greatly limiting the number of processes to look at.
That's an interesting thought. Do you have a trick for doing this within an init script? I would only expect a modest improvement though. I tend to see few processes beyond condor_master children on most execute nodes, and the primary cause of process explosion on submit nodes is condor_master children.
Actually makes me wonder how configurable the proc polling interval is for the master.
However, it's a long road to getting rid of the procd. Right now, this is all another tool in the procd's arsenal. I'd want to see these techniques "work well" in the wild, before we think about disabling parts of the procd, then slowly disabling functionality if the cgroups take care of it. For example, it shouldn't be necessary to scan all the PIDs anymore as the kernel keeps the hierarchies. There's so much procd code that I don't see a clear way to do a "big bang" replacement (not to mention that "legacy" linuxes, such as basically all the deployed Condor installs, will take a long time to phase out). If we head down this direction successfully, maybe we can get rid of it in 5 years?
I'm going to take that to mean that resources aren't aggregated for the namespace. 8o)
Good thing they are for the cgroup ammo. Best, matt