Re: [HTCondor-users] Limiting HTCondor total RAM usage

[Brian Candler <b.candler@xxxxxxxxx>]

On 24/02/2015 16:07, Greg Thain wrote:
> > Unfortunately, I spoke to soon. What happened then is that oom_killer 
> > started killing processes when I don't think it should.
> >
> > I can cat memory.usage_in_bytes every second and I see it slowly 
> > creeping up to 1.6G or 1.7G, and then suddenly the process dies, and 
> > dmesg shows a splurge of backtrace of oom_killer output.
>
> Can you share the specific dmesg output about the oom killer?
Sure. One is below. Note: these jobs do a lot of writing (including 
mainly output to NFS), and there might be a lot of dirty pages in the 
cache to be flushed, which is what the backtrace suggests to me. Do 
dirty VFS pages count towards a cgroup's memory utilisation?

However what I don't understand is why I get oom killers with this config:

BASE_CGROUP = htcondor
CGROUP_MEMORY_LIMIT_POLICY = none

but get no oom killers when those two lines are removed. The only 
difference then is whether condor creates cgroups for its slots, but 
they should be non-enforcing cgroups.

Regards,

Brian.


[  665.666724] decompressqsi invoked oom-killer: gfp_mask=0x200d2, 
order=0, oom_adj=0, oom_score_adj=16
[  665.666728] decompressqsi 
cpuset=condor_var_lib_condor_execute_slot1_24@xxxxxxxxxxxxxxxxxxxx 
mems_allowed=0
[  665.666731] Pid: 7146, comm: decompressqsi Not tainted 3.2.0-4-amd64 
#1 Debian 3.2.65-1+deb7u1
[  665.666733] Call Trace:
[  665.666742]  [<ffffffff810b787a>] ? dump_header+0x78/0x1bd
[  665.666748]  [<ffffffff81350e3f>] ? _raw_spin_unlock_irqrestore+0xe/0xf
[  665.666752]  [<ffffffff8109817e>] ? delayacct_end+0x72/0x7d
[  665.666756]  [<ffffffff811654d2>] ? 
security_real_capable_noaudit+0x40/0x4f
[  665.666761]  [<ffffffff81087760>] ? 
__cpuset_node_allowed_hardwall+0x47/0x50
[  665.666764]  [<ffffffff81350e3f>] ? _raw_spin_unlock_irqrestore+0xe/0xf
[  665.666767]  [<ffffffff810b7c4f>] ? oom_kill_process+0x49/0x271
[  665.666769]  [<ffffffff810b834a>] ? out_of_memory+0x2ea/0x337
[  665.666773]  [<ffffffff810bbfe2>] ? __alloc_pages_nodemask+0x62a/0x7ab
[  665.666777]  [<ffffffff81027cf3>] ? __x2apic_send_IPI_mask+0x116/0x13b
[  665.666782]  [<ffffffff810e5b2b>] ? alloc_pages_current+0xc7/0xe4
[  665.666786]  [<ffffffff81036618>] ? should_resched+0x5/0x23
[  665.666790]  [<ffffffff81102200>] ? pipe_write+0x23e/0x475
[  665.666794]  [<ffffffff810cef9d>] ? set_pmd+0x5/0x8
[  665.666798]  [<ffffffff810facc8>] ? do_sync_write+0xb4/0xec
[  665.666801]  [<ffffffff811653c5>] ? security_file_permission+0x16/0x2d
[  665.666804]  [<ffffffff810fb3b9>] ? vfs_write+0xa2/0xe9
[  665.666806]  [<ffffffff810fb596>] ? sys_write+0x45/0x6b
[  665.666810]  [<ffffffff81355ed2>] ? system_call_fastpath+0x16/0x1b
[  665.666811] Mem-Info:
[  665.666812] Node 0 DMA per-cpu:
[  665.666815] CPU    0: hi:    0, btch:   1 usd:   0
[  665.666816] CPU    1: hi:    0, btch:   1 usd:   0
[  665.666817] CPU    2: hi:    0, btch:   1 usd:   0
[  665.666819] CPU    3: hi:    0, btch:   1 usd:   0
[  665.666820] CPU    4: hi:    0, btch:   1 usd:   0
[  665.666822] CPU    5: hi:    0, btch:   1 usd:   0
[  665.666823] CPU    6: hi:    0, btch:   1 usd:   0
[  665.666824] CPU    7: hi:    0, btch:   1 usd:   0
[  665.666825] CPU    8: hi:    0, btch:   1 usd:   0
[  665.666827] CPU    9: hi:    0, btch:   1 usd:   0
[  665.666828] CPU   10: hi:    0, btch:   1 usd:   0
[  665.666829] CPU   11: hi:    0, btch:   1 usd:   0
[  665.666830] CPU   12: hi:    0, btch:   1 usd:   0
[  665.666832] CPU   13: hi:    0, btch:   1 usd:   0
[  665.666833] CPU   14: hi:    0, btch:   1 usd:   0
[  665.666834] CPU   15: hi:    0, btch:   1 usd:   0
[  665.666835] CPU   16: hi:    0, btch:   1 usd:   0
[  665.666837] CPU   17: hi:    0, btch:   1 usd:   0
[  665.666838] CPU   18: hi:    0, btch:   1 usd:   0
[  665.666839] CPU   19: hi:    0, btch:   1 usd:   0
[  665.666841] CPU   20: hi:    0, btch:   1 usd:   0
[  665.666842] CPU   21: hi:    0, btch:   1 usd:   0
[  665.666843] CPU   22: hi:    0, btch:   1 usd:   0
[  665.666845] CPU   23: hi:    0, btch:   1 usd:   0
[  665.666846] CPU   24: hi:    0, btch:   1 usd:   0
[  665.666847] CPU   25: hi:    0, btch:   1 usd:   0
[  665.666848] CPU   26: hi:    0, btch:   1 usd:   0
[  665.666850] CPU   27: hi:    0, btch:   1 usd:   0
[  665.666851] CPU   28: hi:    0, btch:   1 usd:   0
[  665.666852] CPU   29: hi:    0, btch:   1 usd:   0
[  665.666853] CPU   30: hi:    0, btch:   1 usd:   0
[  665.666855] CPU   31: hi:    0, btch:   1 usd:   0
[  665.666856] Node 0 DMA32 per-cpu:
[  665.666858] CPU    0: hi:  186, btch:  31 usd:   0
[  665.666859] CPU    1: hi:  186, btch:  31 usd:   0
[  665.666860] CPU    2: hi:  186, btch:  31 usd:   0
[  665.666862] CPU    3: hi:  186, btch:  31 usd:   0
[  665.666863] CPU    4: hi:  186, btch:  31 usd:   0
[  665.666864] CPU    5: hi:  186, btch:  31 usd:   0
[  665.666865] CPU    6: hi:  186, btch:  31 usd:   0
[  665.666867] CPU    7: hi:  186, btch:  31 usd:   0
[  665.666868] CPU    8: hi:  186, btch:  31 usd:   0
[  665.666869] CPU    9: hi:  186, btch:  31 usd:   0
[  665.666871] CPU   10: hi:  186, btch:  31 usd:   0
[  665.666872] CPU   11: hi:  186, btch:  31 usd:   0
[  665.666873] CPU   12: hi:  186, btch:  31 usd:   0
[  665.666874] CPU   13: hi:  186, btch:  31 usd:   0
[  665.666876] CPU   14: hi:  186, btch:  31 usd:   0
[  665.666877] CPU   15: hi:  186, btch:  31 usd:   0
[  665.666878] CPU   16: hi:  186, btch:  31 usd:   0
[  665.666880] CPU   17: hi:  186, btch:  31 usd:  30
[  665.666881] CPU   18: hi:  186, btch:  31 usd:   0
[  665.666882] CPU   19: hi:  186, btch:  31 usd:   0
[  665.666884] CPU   20: hi:  186, btch:  31 usd:   0
[  665.666885] CPU   21: hi:  186, btch:  31 usd:   0
[  665.666886] CPU   22: hi:  186, btch:  31 usd:   0
[  665.666887] CPU   23: hi:  186, btch:  31 usd:   0
[  665.666889] CPU   24: hi:  186, btch:  31 usd:   0
[  665.666890] CPU   25: hi:  186, btch:  31 usd:   0
[  665.666891] CPU   26: hi:  186, btch:  31 usd:   0
[  665.666892] CPU   27: hi:  186, btch:  31 usd:   0
[  665.666894] CPU   28: hi:  186, btch:  31 usd:   0
[  665.666895] CPU   29: hi:  186, btch:  31 usd:   0
[  665.666896] CPU   30: hi:  186, btch:  31 usd:   0
[  665.666898] CPU   31: hi:  186, btch:  31 usd:   0
[  665.666898] Node 0 Normal per-cpu:
[  665.666900] CPU    0: hi:  186, btch:  31 usd:   3
[  665.666901] CPU    1: hi:  186, btch:  31 usd:   0
[  665.666903] CPU    2: hi:  186, btch:  31 usd:   3
[  665.666904] CPU    3: hi:  186, btch:  31 usd:   3
[  665.666905] CPU    4: hi:  186, btch:  31 usd:   1
[  665.666906] CPU    5: hi:  186, btch:  31 usd:   3
[  665.666908] CPU    6: hi:  186, btch:  31 usd:   0
[  665.666909] CPU    7: hi:  186, btch:  31 usd:   2
[  665.666910] CPU    8: hi:  186, btch:  31 usd:   2
[  665.666911] CPU    9: hi:  186, btch:  31 usd:   0
[  665.666912] CPU   10: hi:  186, btch:  31 usd:   0
[  665.666914] CPU   11: hi:  186, btch:  31 usd:  30
[  665.666915] CPU   12: hi:  186, btch:  31 usd:   2
[  665.666916] CPU   13: hi:  186, btch:  31 usd:   4
[  665.666918] CPU   14: hi:  186, btch:  31 usd:   2
[  665.666919] CPU   15: hi:  186, btch:  31 usd:   4
[  665.666920] CPU   16: hi:  186, btch:  31 usd:   0
[  665.666922] CPU   17: hi:  186, btch:  31 usd:  29
[  665.666923] CPU   18: hi:  186, btch:  31 usd:   0
[  665.666924] CPU   19: hi:  186, btch:  31 usd:   0
[  665.666926] CPU   20: hi:  186, btch:  31 usd:   0
[  665.666927] CPU   21: hi:  186, btch:  31 usd:   4
[  665.666928] CPU   22: hi:  186, btch:  31 usd:   2
[  665.666929] CPU   23: hi:  186, btch:  31 usd:   2
[  665.666931] CPU   24: hi:  186, btch:  31 usd:   1
[  665.666932] CPU   25: hi:  186, btch:  31 usd:   1
[  665.666933] CPU   26: hi:  186, btch:  31 usd:   0
[  665.666934] CPU   27: hi:  186, btch:  31 usd:   2
[  665.666936] CPU   28: hi:  186, btch:  31 usd:   1
[  665.666937] CPU   29: hi:  186, btch:  31 usd:   3
[  665.666938] CPU   30: hi:  186, btch:  31 usd:  32
[  665.666940] CPU   31: hi:  186, btch:  31 usd:   1
[  665.666944] active_anon:8008254 inactive_anon:43 isolated_anon:0
[  665.666944]  active_file:195745 inactive_file:92438 isolated_file:363
[  665.666945]  unevictable:0 dirty:0 writeback:0 unstable:110
[  665.666946]  free:7922054 slab_reclaimable:11805 slab_unreclaimable:84666
[  665.666946]  mapped:3480 shmem:87 pagetables:21183 bounce:0
[  665.666948] Node 0 DMA free:15892kB min:32kB low:40kB high:48kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15668kB 
mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB 
pagetables:0kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 
all_unreclaimable? yes
[  665.666955] lowmem_reserve[]: 0 1949 32249 32249
[  665.666958] Node 0 DMA32 free:125220kB min:4076kB low:5092kB 
high:6112kB active_anon:1833060kB inactive_anon:0kB active_file:64kB 
inactive_file:1160kB unevictable:0kB isolated(anon):0kB 
isolated(file):0kB present:1995876kB mlocked:0kB dirty:12kB 
writeback:0kB mapped:32kB shmem:0kB slab_reclaimable:1324kB 
slab_unreclaimable:14180kB kernel_stack:64kB pagetables:3700kB 
unstable:12kB bounce:0kB writeback_tmp:0kB pages_scanned:3580 
all_unreclaimable? yes
[  665.666966] lowmem_reserve[]: 0 0 30300 30300
[  665.666968] Node 0 Normal free:62940kB min:63412kB low:79264kB 
high:95116kB active_anon:30152284kB inactive_anon:92kB 
active_file:12196kB inactive_file:16544kB unevictable:0kB 
isolated(anon):0kB isolated(file):1452kB present:31027200kB mlocked:0kB 
dirty:0kB writeback:132kB mapped:1712kB shmem:216kB 
slab_reclaimable:13596kB slab_unreclaimable:269052kB kernel_stack:2280kB 
pagetables:62312kB unstable:428kB bounce:0kB writeback_tmp:0kB 
pages_scanned:59998 all_unreclaimable? yes
[  665.666976] lowmem_reserve[]: 0 0 0 0
[  665.666978] Node 0 DMA: 1*4kB 0*8kB 1*16kB 0*32kB 2*64kB 1*128kB 
1*256kB 0*512kB 1*1024kB 1*2048kB 3*4096kB = 15892kB
[  665.666984] Node 0 DMA32: 80*4kB 205*8kB 141*16kB 106*32kB 248*64kB 
73*128kB 11*256kB 1*512kB 5*1024kB 1*2048kB 20*4096kB = 125240kB
[  665.666990] Node 0 Normal: 2533*4kB 1113*8kB 1000*16kB 308*32kB 
215*64kB 43*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB = 68252kB
[  665.666996] 287658 total pagecache pages
[  665.666997] 0 pages in swap cache
[  665.666998] Swap cache stats: add 0, delete 0, find 0/0
[  665.666999] Free swap  = 0kB
[  665.667000] Total swap = 0kB
[  665.848067] 16777200 pages RAM
[  665.848069] 255489 pages reserved
[  665.848070] 367434 pages shared
[  665.848071] 8260659 pages non-shared
[  665.848072] [ pid ]   uid  tgid total_vm      rss cpu oom_adj 
oom_score_adj name
[  665.848110] [  861]     0   861     5475      460 21     -17         
-1000 udevd
[  665.848113] [ 1321]     0  1321     5474      346 18     -17         
-1000 udevd
[  665.848115] [ 1324]     0  1324     5474      347 19     -17         
-1000 udevd
[  665.848119] [ 2829]     0  2829     4744      212 18       
0             0 rpcbind
[  665.848122] [ 2840]   102  2840     5837      324 2       
0             0 rpc.statd
[  665.848124] [ 2862]     0  2862     6351      244 21       
0             0 rpc.idmapd
[  665.848127] [ 3390]     0  3390    29743      582 0       
0             0 rsyslogd
[  665.848129] [ 3484]     0  3484     3272      154 25       
0             0 mdadm
[  665.848132] [ 3522]     0  3522     4169       38 0       
0             0 atd
[  665.848134] [ 3570]     0  3570     5103      260 18       
0             0 cron
[  665.848136] [ 3628]     0  3628     1030      151 8       
0             0 acpid
[  665.848138] [ 3648]   106  3648     7452       71 25       
0             0 dbus-daemon
[  665.848140] [ 3843]   104  3843    12144     1270 1       
0             0 snmpd
[  665.848142] [ 3884]     0  3884    12484      282 11     -17         
-1000 sshd
[  665.848144] [ 3916]   101  3916    11704      247 0       
0             0 exim4
[  665.848147] [ 3952]     0  3952    17815      548 1       
0             0 sshd
[  665.848149] [ 3959]     0  3959     5161      887 13       
0             0 bash
[  665.848151] [ 4068]   104  4068     2393      448 1       
0             0 snmpd-mdraid-co
[  665.848153] [ 4127]   107  4127     9768      428 18       
0             0 ntpd
[  665.848155] [ 4155]     0  4155     4065      226 8       
0             0 getty
[  665.848157] [ 4156]     0  4156     4065      225 1       
0             0 getty
[  665.848159] [ 4157]     0  4157     4065      226 9       
0             0 getty
[  665.848161] [ 4158]     0  4158     4065      228 28       
0             0 getty
[  665.848163] [ 4159]     0  4159     4065      226 0       
0             0 getty
[  665.848165] [ 4160]     0  4160     4065      227 11       
0             0 getty
[  665.848167] [ 5485]     0  5485    24169      797 28       
0             0 condor_master
[  665.848169] [ 5486]     0  5486     5958     1481 6       
0             0 condor_procd
[  665.848171] [ 5487]     0  5487    24965     1839 22       
0             0 condor_startd
[  665.848174] [ 6073]     0  6073    17815      885 1       
0             0 sshd
[  665.848176] [ 6077]     0  6077     5180      911 8       
0             0 bash
[  665.848178] [ 6159]   104  6159     2398      474 19       
0             0 snmpd-smartctl-
[  665.848181] [ 6947]     0  6947    23930     1759 4       0           
-16 condor_starter
[  665.848183] [ 6957]  1012  6957    20792    11282 28       
0            16 python
[  665.848185] [ 6977]     0  6977    23930     1762 5       0           
-16 condor_starter
[  665.848187] [ 6986]     0  6986    23930     1762 7       0           
-16 condor_starter
[  665.848189] [ 6995]  1012  6995    20792    11299 4       
0            16 python
[  665.848191] [ 7004]  1012  7004    20792    11299 1       
0            16 python
[  665.848201] [ 7007]  1012  7007     4415      367 19       
0            16 bash
[  665.848204] [ 7008]  1012  7008   213460   205852 9       
0            16 filterzqsi
[  665.848206] [ 7009]  1012  7009    32833      252 15       
0            16 decompressqsi
[  665.848208] [ 7010]  1012  7010   238619   172724 11       
0            16 qsitorsiandspli
[  665.848210] [ 7015]  1012  7015     4415      366 22       
0            16 bash
[  665.848212] [ 7016]  1012  7016     4415      367 6       
0            16 bash
[  665.848214] [ 7017]  1012  7017   213460   205828 21       
0            16 filterzqsi
[  665.848217] [ 7018]  1012  7018    32833      291 2       
0            16 decompressqsi
[  665.848218] [ 7019]  1012  7019   238619   214757 3       
0            16 qsitorsiandspli
[  665.848220] [ 7020]  1012  7020   213460   205813 15       
0            16 filterzqsi
[  665.848223] [ 7021]  1012  7021    32833      289 8       
0            16 decompressqsi
[  665.848225] [ 7022]  1012  7022   238619   215675 11       
0            16 qsitorsiandspli
[  665.848227] [ 7026]     0  7026    23930     1763 13       
0           -16 condor_starter
[  665.848229] [ 7035]  1012  7035    20792    11302 15       
0            16 python
[  665.848231] [ 7045]     0  7045    23930     1765 16       
0           -16 condor_starter
[  665.848233] [ 7058]  1012  7058    20792    11301 3       
0            16 python
[  665.848235] [ 7061]  1012  7061     4415      366 25       
0            16 bash
[  665.848237] [ 7062]  1012  7062   213460   205827 23       
0            16 filterzqsi
[  665.848239] [ 7063]  1012  7063    32833      259 7       
0            16 decompressqsi
[  665.848241] [ 7064]  1012  7064   238619    82527 0       
0            16 qsitorsiandspli
[  665.848243] [ 7066]  1012  7066     4415      366 18       
0            16 bash
[  665.848245] [ 7067]  1012  7067   213460   205865 6       
0            16 filterzqsi
[  665.848247] [ 7068]  1012  7068    32833      257 18       
0            16 decompressqsi
[  665.848249] [ 7069]  1012  7069   238619   161585 18       
0            16 qsitorsiandspli
[  665.848251] [ 7072]     0  7072    23930     1765 6       0           
-16 condor_starter
[  665.848262] [ 7081]  1012  7081    20792    11301 11       
0            16 python
[  665.848264] [ 7100]  1012  7100     4415      366 24       
0            16 bash
[  665.848266] [ 7101]  1012  7101   213460   205798 12       
0            16 filterzqsi
[  665.848268] [ 7102]  1012  7102    32833      261 10       
0            16 decompressqsi
[  665.848270] [ 7103]  1012  7103   238619   182830 20       
0            16 qsitorsiandspli
[  665.848272] [ 7109]     0  7109    23930     1766 16       
0           -16 condor_starter
[  665.848274] [ 7117]     0  7117    23930     1765 0       0           
-16 condor_starter
[  665.848277] [ 7135]  1012  7135    20792    11301 11       
0            16 python
[  665.848279] [ 7136]  1012  7136    20792    11300 15       
0            16 python
[  665.848281] [ 7144]  1012  7144     4415      367 26       
0            16 bash
[  665.848282] [ 7145]  1012  7145   213460   205864 25       
0            16 filterzqsi
[  665.848284] [ 7146]  1012  7146    32833      273 9       
0            16 decompressqsi
[  665.848286] [ 7147]  1012  7147   238619   134726 8       
0            16 qsitorsiandspli
[  665.848288] [ 7152]  1012  7152     4415      366 30       
0            16 bash
[  665.848290] [ 7153]  1012  7153   213460   205813 2       
0            16 filterzqsi
[  665.848292] [ 7154]  1012  7154    32833      265 3       
0            16 decompressqsi
[  665.848294] [ 7155]  1012  7155   238619   135585 5       
0            16 qsitorsiandspli
[  665.848296] [ 7156]     0  7156    23930     1765 16       
0           -16 condor_starter
[  665.848298] [ 7163]     0  7163    23930     1765 18       
0           -16 condor_starter
[  665.848300] [ 7172]  1012  7172    20792    11302 8       
0            16 python
[  665.848302] [ 7175]     0  7175    23930     1766 0       0           
-16 condor_starter
[  665.848304] [ 7176]  1012  7176    20792    11301 14       
0            16 python
[  665.848306] [ 7193]  1012  7193    20792    11303 8       
0            16 python
[  665.848308] [ 7197]  1012  7197     4415      366 29       
0            16 bash
[  665.848310] [ 7198]  1012  7198   213460   205817 4       
0            16 filterzqsi
[  665.848312] [ 7199]  1012  7199    32833      268 22       
0            16 decompressqsi
[  665.848314] [ 7200]  1012  7200   238619   108655 0       
0            16 qsitorsiandspli
[  665.848316] [ 7201]  1012  7201     4415      367 31       
0            16 bash
[  665.848318] [ 7202]  1012  7202   213460   205808 6       
0            16 filterzqsi
[  665.848320] [ 7203]  1012  7203    32833      287 2       
0            16 decompressqsi
[  665.848322] [ 7204]  1012  7204   238619   110709 3       
0            16 qsitorsiandspli
[  665.848324] [ 7207]  1012  7207     4415      366 28       
0            16 bash
[  665.848325] [ 7208]  1012  7208   213460   205832 9       
0            16 filterzqsi
[  665.848327] [ 7209]  1012  7209    32833      295 15       
0            16 decompressqsi
[  665.848329] [ 7210]  1012  7210   238619    79891 10       
0            16 qsitorsiandspli
[  665.848332] [ 7211]     0  7211    23930     1767 0       0           
-16 condor_starter
[  665.848334] [ 7224]  1012  7224    20792    11301 11       
0            16 python
[  665.848335] [ 7230]     0  7230    23930     1766 0       0           
-16 condor_starter
[  665.848337] [ 7243]  1012  7243    20792    11302 1       
0            16 python
[  665.848339] [ 7249]  1012  7249     4415      367 18       
0            16 bash
[  665.848341] [ 7250]  1012  7250   213460   205829 3       
0            16 filterzqsi
[  665.848343] [ 7251]  1012  7251    32833      283 6       
0            16 decompressqsi
[  665.848345] [ 7252]  1012  7252   238619    70850 29       
0            16 qsitorsiandspli
[  665.848347] [ 7253]     0  7253    23930     1767 0       0           
-16 condor_starter
[  665.848350] [ 7264]  1012  7264    20792    11298 9       
0            16 python
[  665.848352] [ 7267]  1012  7267     4415      367 16       
0            16 bash
[  665.848354] [ 7268]  1012  7268   213460   205844 15       
0            16 filterzqsi
[  665.848356] [ 7269]  1012  7269    32833      301 13       
0            16 decompressqsi
[  665.848358] [ 7270]  1012  7270   238619    83781 29       
0            16 qsitorsiandspli
[  665.848360] [ 7274]     0  7274    23930     1766 6       0           
-16 condor_starter
[  665.848362] [ 7283]  1012  7283    20792    11301 4       
0            16 python
[  665.848364] [ 7299]  1012  7299     4415      366 31       
0            16 bash
[  665.848366] [ 7301]  1012  7301   213460   205828 4       
0            16 filterzqsi
[  665.848368] [ 7302]  1012  7302    32833      283 0       
0            16 decompressqsi
[  665.848370] [ 7303]  1012  7303   238619    84577 2       
0            16 qsitorsiandspli
[  665.848372] [ 7306]     0  7306    23930     1765 1       0           
-16 condor_starter
[  665.848374] [ 7315]  1012  7315    20792    11301 4       
0            16 python
[  665.848376] [ 7325]  1012  7325     4415      366 18       
0            16 bash
[  665.848378] [ 7326]  1012  7326   213460   205819 14       
0            16 filterzqsi
[  665.848380] [ 7327]  1012  7327    32833      296 10       
0            16 decompressqsi
[  665.848382] [ 7328]  1012  7328   238619    84308 9       
0            16 qsitorsiandspli
[  665.848384] [ 7331]     0  7331    23930     1767 5       0           
-16 condor_starter
[  665.848386] [ 7340]  1012  7340    20792    11303 1       
0            16 python
[  665.848388] [ 7343]  1012  7343     4415      367 18       
0            16 bash
[  665.848390] [ 7344]  1012  7344   213460   205819 23       
0            16 filterzqsi
[  665.848392] [ 7345]  1012  7345    32833      268 4       
0            16 decompressqsi
[  665.848394] [ 7346]  1012  7346   238619    84379 0       
0            16 qsitorsiandspli
[  665.848396] [ 7347]  1012  7347     4415      366 22       
0            16 bash
[  665.848398] [ 7348]  1012  7348   213460   205829 16       
0            16 filterzqsi
[  665.848400] [ 7349]  1012  7349    32833      300 1       
0            16 decompressqsi
[  665.848402] [ 7350]  1012  7350   238619   109918 2       
0            16 qsitorsiandspli
[  665.848404] [ 7362]     0  7362    23930     1767 6       0           
-16 condor_starter
[  665.848406] [ 7371]  1012  7371    20792    11303 9       
0            16 python
[  665.848408] [ 7379]  1012  7379     4415      366 29       
0            16 bash
[  665.848410] [ 7380]  1012  7380   213460   205840 30       
0            16 filterzqsi
[  665.848412] [ 7381]  1012  7381    32833      285 30       
0            16 decompressqsi
[  665.848413] [ 7382]  1012  7382   238619    40424 10       
0            16 qsitorsiandspli
[  665.848415] [ 7386]     0  7386    23930     1767 0       0           
-16 condor_starter
[  665.848417] [ 7387]     0  7387    23930     1766 2       0           
-16 condor_starter
[  665.848419] [ 7402]  1012  7402    20792    11300 10       
0            16 python
[  665.848421] [ 7403]  1012  7403    20792    11301 12       
0            16 python
[  665.848423] [ 7418]  1012  7418     4415      365 27       
0            16 bash
[  665.848425] [ 7419]  1012  7419     4415      357 25       
0            16 bash
[  665.848427] [ 7421]  1012  7421   213460   205820 6       
0            16 filterzqsi
[  665.848429] [ 7420]  1012  7420   213460   205846 11       
0            16 filterzqsi
[  665.848431] [ 7422]  1012  7422    32833      291 1       
0            16 decompressqsi
[  665.848433] [ 7423]  1012  7423    32833      248 10       
0            16 decompressqsi
[  665.848435] [ 7424]  1012  7424   238619   101797 26       
0            16 qsitorsiandspli
[  665.848437] [ 7425]  1012  7425   238619    54217 24       
0            16 qsitorsiandspli
[  665.848439] [ 7432]     0  7432    23930     1767 1       0           
-16 condor_starter
[  665.848441] [ 7437]     0  7437    23930     1766 0       0           
-16 condor_starter
[  665.848442] [ 7450]  1012  7450    20792    11302 8       
0            16 python
[  665.848444] [ 7451]  1012  7451    20792    11302 9       
0            16 python
[  665.848446] [ 7471]     0  7471    23930     1766 6       0           
-16 condor_starter
[  665.848448] [ 7480]  1012  7480     4415      367 25       
0            16 bash
[  665.848450] [ 7481]  1012  7481     4415      367 24       
0            16 bash
[  665.848452] [ 7482]  1012  7482   213460   205841 2       
0            16 filterzqsi
[  665.848454] [ 7483]  1012  7483    32833      305 17       
0            16 decompressqsi
[  665.848456] [ 7484]  1012  7484   238619    52699 2       
0            16 qsitorsiandspli
[  665.848458] [ 7485]  1012  7485   167310   159609 4       
0            16 filterzqsi
[  665.848460] [ 7486]  1012  7486    32832      284 11       
0            16 decompressqsi
[  665.848462] [ 7487]  1012  7487   238619    33229 8       
0            16 qsitorsiandspli
[  665.848464] [ 7490]  1012  7490    20792    11302 20       
0            16 python
[  665.848466] [ 7497]     0  7497    23930     1765 0       0           
-16 condor_starter
[  665.848468] [ 7504]     0  7504    23930     1765 0       0           
-16 condor_starter
[  665.848470] [ 7511]  1012  7511     4415      367 26       
0            16 bash
[  665.848472] [ 7512]  1012  7512   149358   141652 8       
0            16 filterzqsi
[  665.848474] [ 7513]  1012  7513    32832      229 2       
0            16 decompressqsi
[  665.848475] [ 7514]  1012  7514   238619    33230 22       
0            16 qsitorsiandspli
[  665.848478] [ 7519]  1012  7519    20792    11302 12       
0            16 python
[  665.848479] [ 7520]  1012  7520    20792    11302 13       
0            16 python
[  665.848481] [ 7535]  1012  7535     4415      367 28       
0            16 bash
[  665.848483] [ 7536]  1012  7536   199287   191556 15       
0            16 filterzqsi
[  665.848485] [ 7537]  1012  7537    32832      280 10       
0            16 decompressqsi
[  665.848487] [ 7538]  1012  7538   238619    33231 1       
0            16 qsitorsiandspli
[  665.848489] [ 7539]  1012  7539     4415      367 25       
0            16 bash
[  665.848491] [ 7540]  1012  7540   123585   115854 21       
0            16 filterzqsi
[  665.848493] [ 7541]  1012  7541    32832      282 29       
0            16 decompressqsi
[  665.848495] [ 7542]  1012  7542   238619    33230 2       
0            16 qsitorsiandspli
[  665.848497] [ 7546]     0  7546    23930     1766 2       0           
-16 condor_starter
[  665.848499] [ 7555]  1012  7555    20791    11298 9       
0            16 python
[  665.848501] [ 7568]  1012  7568     4415      367 18       
0            16 bash
[  665.848503] [ 7569]  1012  7569   213204   205250 0       
0            16 filterzqsi
[  665.848505] [ 7570]  1012  7570    32832      280 3       
0            16 decompressqsi
[  665.848507] [ 7571]  1012  7571   238619    33238 5       
0            16 qsitorsiandspli
[  665.848509] [ 7575]     0  7575    23930     1766 1       0           
-16 condor_starter
[  665.848511] [ 7584]  1012  7584    19645     9655 12       
0            16 python
[  665.848513] [ 7594]     0  7594    23930     1766 1       0           
-16 condor_starter
[  665.848515] [ 7603]  1012  7603    20792    11340 14       
0            16 python
[  665.848517] [ 7607]     0  7607    23930     1766 22       
0           -16 condor_starter
[  665.848519] [ 7614]  1012  7614     4415      366 26       
0            16 bash
[  665.848521] [ 7615]  1012  7615    37752    30006 27       
0            16 filterzqsi
[  665.848522] [ 7616]  1012  7616    32832      245 26       
0            16 decompressqsi
[  665.848524] [ 7617]  1012  7617    39631      963 27       
0            16 qsitorsiandspli
[  665.848526] [ 7618]  1012  7618      796        1 6       
0            16 python