Re: [HTCondor-users] increasing schedd memory usage [v8.6.0?]

[Thomas Hartmann <thomas.hartmann@xxxxxxx>]

Hi Brian,

thanks for the suggestion

On 2017-02-10 03:19, Brian Bockelman wrote:
> Is it possible the extra memory usage is coming from when the condor_schedd process forks to respond to condor_q queries?  Are you seeing an abnormally large amount of queries?

not that I am aware of - any queries would should come only from the ARC
CE, but afais both our ARCCEs have been ~equally busy.
As cross-check, I restarted the CE daemon, but it had no effect on the
memory consumption so far and only reduced the number of connections to
the outside [1] compared to its sibling (should be the expected behaviour).
On the affected node quite(?) a number of shadows were kept open [2],
but that should be OK, or?

We have now downgraded the version to
  8.4.11
and will keep an eye on it over the weekend.
If the behaviour gets back to normal, we can at least exclude Condor.

Cheers,
  Thomas




[1]
> grid-arcce1 > wc -l /proc/net/tcp*
335 /proc/net/tcp
10 /proc/net/tcp6
345 total

> grid-arcce0 > wc -l /proc/net/tcp*
2733 /proc/net/tcp
16 /proc/net/tcp6
2749 total


[2]
> lsof -i TCP | grep condor | cut -d " " -f 1 | sort | uniq -c
      1 condor_de
      1 condor_ma
      4 condor_sc
> lsof | grep condor | cut -d " " -f 1 | sort | uniq -c
     27 condor_de
     30 condor_ma
     19 condor_pr
     45 condor_sc
  44776 condor_sh
      1 scan-cond

Attachment: smime.p7s
Description: S/MIME Cryptographic Signature