Re: [HTCondor-users] increasing schedd memory usage [v8.6.0?]

[Brian Bockelman <bbockelm@xxxxxxxxxxx>]

Hi Thomas,

I can't think of anything that would cause this.

Is it possible the extra memory usage is coming from when the condor_schedd process forks to respond to condor_q queries?  Are you seeing an abnormally large amount of queries?

Brian

> On Feb 9, 2017, at 11:04 AM, Thomas Hartmann <thomas.hartmann@xxxxxxx> wrote:
> 
> Hi all,
> 
> we are observing since yesterday a suddenly increasing memory usage on
> one of our condor schedulers [1].
> Several condor schedds processes start allocating each ~35% of memory
> (16gB overall). Judging from the (coarse grained (1m)) monitoring,
> memory usage is oscillating with ~5m between 50% and >100% and swapping.
> After restarting the service/node, memory consumption got down
> significantly but peaked again later during the day.
> 
> So far we have not found a cause for the sudden hunger for memory.
> 
> On Feb. 01 we upgraded the node from 8.4.1 to 8.6.1 and kept its sibling
> node on 8.4.1. So far, the 8.4.1 sibling has not shown the behaviour. On
> the other hand it took the 8.6.1 node more than week to get into the
> current state, so it is imho not very conclusive to suggest the update
> as cause? [2]
> 
> Maybe somebody has an idea, where to look for clues?
> 
> Cheers and thanks for ideas,
>  Thomas
> 
> 
> [1]
> grid-arcce1.desy.de - Nordugrid ARC CE submitting to Condor
> 
>> free
>             total       used       free     shared    buffers     cached
> Mem:      16268636    8605212    7663424        308     423620     551716
> -/+ buffers/cache:    7629876    8638760
> Swap:      4095996    1308184    2787812
> 
> [2]
> in the release notes I found nothing matching that could point to our
> observations
> http://research.cs.wisc.edu/htcondor/manual/v8.6/10_3Stable_Release.html
> 
> <grid-arcce1_mem_3d_201702091735.png>_______________________________________________
> HTCondor-users mailing list
> To unsubscribe, send a message to htcondor-users-request@xxxxxxxxxxx with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users
> 
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/htcondor-users/