On Jul 17, 2012, at 6:37 PM, Brian Bockelman wrote:
Hi,
When I last talked to Miron about multi-shadow, he suggested first wringing every last byte out of the current one before even proposing the multi-shadow. So, I spent about an hour with igprof and staring at smaps.
I measured a shadow as having 360KB of heap, about 550KB total unshared space, and 274KB of data live on the heap (so about 25% waste due to fragmentation).
Here's what I found that we could save. List is in ascending order of difficulty to implement.
0) Turn off classad caching: 55KB.
1) Copy of job's classad inside the file transfer object: 8KB
2) gethostbyaddr -> gethostbyaddr_r (including all callsites, even in the logging code! See ExecuteEvent::writeEvent): 5KB.
3) getpwnam, getpwuid to reentrant versions: 2KB
4) Remove stats object from DaemonCore for shadow: 7KB
5) libcondor_utils has 156KB of dirty writable memory (non-const statics?) that can't be shared: 100KB? This part was not included in my heap calculations, but is indeed non-shared.
6) Cleanup of auth code to reduce heap fragmentation: 5-15KB
7) Un-loading the IpVerify table after usage: 9KB.
8) The configuration subsystem. This would be one tough nugget to crack (note: would all be shared with the multi-shadow), but is very lightly used after the shadow fires up. 70KB.
Lessons learned:
- Classad caching does more harm than good for a single shadow (20% of heap)
- If we squeeze really hard at odds-n-ends in the heap, we can shrink the heap by 10%. I don't think all the items listed above are plausible (especially 8).
- Non-const globals in libcondor_utils consist of 25% of the total memory footprint. There are 332 source files in libcondor_utils - whack-a-mole time?
- Similarly, there are a few things sitting around in the other Condor libraries, but nothing as sizable.
- Obviously sharable resources for the multi-shadow (parameter subsystem, auth hash maps and tables, daemon core object) make up 50% of the heap.
- It's not immediately obvious how much the ClassAd cache will affect the multi-shadow, but I would expect a bit of sharing. Let's estimate 50% of the current cache is sharable, or 10% of the total heap.
So, we can squeeze about 15% of the shadow size by continuing to shave things and turning off caching.
Assuming 10 jobs per 1 shadow, we could realize a 60% memory gain.
Both numbers become more dramatic if we can figure out who's hanging out in the data segment.
Brian
PS - all numbers have been rounded and self-consistency is limited to my ability to do mental math.
PPS - after 5 minutes with 'nm', it appears the data segment consists primarily of the parameter table. DOH!
_______________________________________________
Condor-devel mailing list
Condor-devel@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/condor-devel
Hi all,
Updated estimates after playing with the shadow for more than an hour:
- All the linked libraries add up a bit more drastically than I originally thought (I had ignored all the small ones and just looked at the big ones in my first estimates). Adding in all my optimizations, there are 888KB of Private_Dirty memory. I squeezed things down to 96KB for libcondor_utils's data segment from 156KB. The savings are more drastic on more recent versions of GCC (where LTO is available). There's not much more left to squeeze out due to vtables, got, and plt.
- There's probably 150KB of unique data per shadow, once you subtract out the things I mention above. Hence, 738KB can be thought of as "overhead". So, running 10 jobs per shadow would result in an 75% memory savings.
In all, it indicates that multi-shadow would be beneficial and "shadow squeezing" is going to continue to have diminishing returns.
Brian
_______________________________________________
Condor-devel mailing list
Condor-devel@xxxxxxxxxxx
https://lists.cs.wisc.edu/mailman/listinfo/condor-devel
