Hi all,
Last night, I took another detour into "cool things in modern Linux kernels" and came up with per-job PID namespaces for Condor:
https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=1959
Basically, when a job runs, the starter requests the a new PID namespace from the kernel. The clone'd process believes it is PID 1, with all process in the job hanging off that. It looks something like this:
[bbockelm@rcf-bockelman condor]$ condor_run ps faux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
bbockelm 1 0.0 0.0 106200 1132 ? SN 14:15 0:00 /bin/bash /home/bbockelm/projects/condor/.condor_run.17238
bbockelm 2 0.0 0.0 108052 1000 ? RN 14:15 0:00 ps faux
However, to the "outside world", these appear as normal processes. The processes inside the job can't view or contact external processes - two jobs running within the same Unix account can't discover or send signals to each other. Additionally, when "PID 1" dies, the kernel wipes out the remaining processes started by the job. It's a fairly neat trick. This all requires kernel 2.6.24 or later.
Enjoy!
Brian