Re: [HTCondor-users] Memory leak in python binding issue still occurring (Ticket #5727)

[Kevin Retzke <kretzke@xxxxxxxx>]

After looking into it a bit, looks the the schedd.query() calls aren't the source of our leaks now. It appears to be coming from the Schedd() initialization and/or collector.locateAll() call. I discovered one of our test probes wasn't actually doing anything due to a authentication issue; it would just try to obtain daemon stats via locateAll() several times, then return with an error. Memory usage went up about 10 KB per call.

Thanks,

Kevin

---

From: Kevin Retzke
Sent: Friday, April 7, 2017 10:55 AM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Memory leak in python binding issue still occurring (Ticket #5727)

 

The memory usage trends for a number of our long-running fifemon probes also seem to suggest a leak still exists  (8.6.1).

http://i.imgur.com/IRP6D2Jl.png

Kevin

---

From: HTCondor-users <htcondor-users-bounces@xxxxxxxxxxx> on behalf of Patil, Jitendra <jitendra.patil@xxxxxxxxx>
Sent: Thursday, April 6, 2017 6:16 PM
To: HTCondor-Users Mail List
Subject: Re: [HTCondor-users] Memory leak in python binding issue still occurring (Ticket #5727)

 

I just tested again with both schedd.query and schedd.xquery (tested on version 8.6.1 and 8.4.11). In both case the memory is increase by 128kb, exactly same behavior as mentioned in original ticket.

 

-Jitendra

 

 

 

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of John M Knoeller
Sent: Thursday, April 06, 2017 3:51 PM
To: HTCondor-Users Mail List <htcondor-users@xxxxxxxxxxx>
Subject: Re: [HTCondor-users] Memory leak in python binding issue still occurring (Ticket #5727)

 

schedd.query and schedd.xquery have completely different implementation - so  there may be a leak with xquery, but it’s not the SAME leak.

 

-tj

 

 

From: HTCondor-users [mailto:htcondor-users-bounces@xxxxxxxxxxx] On Behalf Of Patil, Jitendra
Sent: Thursday, April 6, 2017 5:14 PM
To: htcondor-users@xxxxxxxxxxx
Subject: [HTCondor-users] Memory leak in python binding issue still occurring (Ticket #5727)

 

Hi,

 

I am trying to verify ‘memory leak in condor_q client’ issue that was fixed in June 2016 :  Ticket #5727 - https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=5727

 

But found out that memory leak is still occurring. I have verified that this issue is still occurring on latest stable version 8.6.1 and also on 8.4.11.

 

I used the sample code provided in the ticket to verify the issue:

 

import resource

import htcondor

import gc

schedd = htcondor.Schedd()

while True:

    schedd.xquery()

    # explicitly run the garbage collector to free up mem not leaked

    gc.collect()

    print('mem use: %s (kb)' % resource.getrusage(resource.RUSAGE_SELF).ru_maxrss)

 

 

Can someone please help if I’m missing something or confirm that fix is not available in the latest stable version?

 

-Jitendra