Re: [Condor-users] 7.8.2 / running out of ports for UDP

["Shrum, Donald C" <DCShrum@xxxxxxxxxxxxx>]

Yes.  In fact I am running with no dns.  

-----Original Message-----
From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Rob de Graaf
Sent: Thursday, September 06, 2012 12:26 PM
To: Condor-Users Mail List
Subject: Re: [Condor-users] 7.8.2 / running out of ports for UDP

Just a guess, but are you running with NO_DNS?

https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=2691,3

On 09/06/2012 06:09 PM, Dan Bradley wrote:
>
> Is there anything interesting in MasterLog?
>
> It may be helpful to strace condor_master.
>
> strace -p <insert-pid-of-master> -o master.strace
>
> You can kill that after it has run for long enough to observe lots of 
> sockets being opened.
>
> What is using port 1980?  Have you configured your collector to use 
> that port?
>
> --Dan
>
> On 9/6/12 11:02 AM, Shrum, Donald C wrote:
>>
>> Hi Dan,
>>
>> The problem persists so flocking was not related.  The condor master 
>> is back up to 28,000+ open UDP ports.
>>
>> Right now I'm just doing a periodic restart
>>
>> [root@condor ~]# condor_status -master -format "%d\n"
>> MonitorSelfRegisteredSocketCount
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> connect: Resource temporarily unavailable
>>
>> [root@condor ~]# service condor restart
>>
>> Shutting down Condor (fast-shutdown mode)...  done.
>>
>> Starting up Condor...    done.
>>
>> [root@condor ~]# condor_status -master -format "%d\n"
>> MonitorSelfRegisteredSocketCount
>>
>> 1
>>
>> Thanks for the help -
>>
>> Don
>>
>> FSU HPC
>>
>> *From:*condor-users-bounces@xxxxxxxxxxx
>> [mailto:condor-users-bounces@xxxxxxxxxxx] *On Behalf Of *Dan Bradley
>> *Sent:* Thursday, September 06, 2012 10:53 AM
>> *To:* condor-users@xxxxxxxxxxx
>> *Subject:* Re: [Condor-users] 7.8.2 / running out of ports for UDP
>>
>> Donald,
>>
>> If you observe this problem again, see what the daemon is reporting 
>> in its ClassAd:
>>
>> condor_status -master -format "%d\n" MonitorSelfRegisteredSocketCount 
>> <insert-hostname-here>
>>
>> What's using port 1980?  The collector?
>>
>> --Dan
>>
>> On 9/6/12 9:28 AM, Shrum, Donald C wrote:
>>
>>     As always, thanks Ian.
>>
>>     We had flocking set up with another University and using the 'this
>>     was one of the last things I touched' trouble shooting method I
>>     just disabled flocking and Condor Connection Brokering 
>> (CCB_ADDRESS)
>>
>>     That may have resolved the problem... we'll see.
>>
>>     --Don
>>
>>     FSU HPC
>>
>>     *From:*condor-users-bounces@xxxxxxxxxxx
>>     <mailto:condor-users-bounces@xxxxxxxxxxx>
>>     [mailto:condor-users-bounces@xxxxxxxxxxx] *On Behalf Of *Ian Chesal
>>     *Sent:* Thursday, September 06, 2012 9:34 AM
>>     *To:* Condor-Users Mail List
>>     *Subject:* Re: [Condor-users] 7.8.2 / running out of ports for 
>> UDP
>>
>>     Donald,
>>
>>     You could switch to TCP for collector updates:
>>
>>     UPDATE_COLLECTOR_WITH_TCP = True
>>
>>     See:
>>     
>> http://research.cs.wisc.edu/condor/manual/v7.6/3_3Configuration.html#
>> 16701
>>
>>     Or even better: switch to using the shared port daemon. This
>>     should help reduce the number of connections needed on any one
>>     machine. See:
>>     
>> http://research.cs.wisc.edu/condor/manual/v7.6/3_7Networking_includes
>> .html#32152
>>
>>     Regards,
>>
>>     - Ian
>>
>>     --
>>
>>     Ian Chesal
>>
>>     Cycle Computing, LLC
>>
>>     Leader in Open Compute Solutions for Clouds, Servers, and 
>> Desktops
>>
>>     Enterprise Condor Support and Management Tools
>>
>>     888.292.5320
>>
>>     http://www.cyclecomputing.com <http://www.cyclecomputing.com/>
>>
>>     http://www.cyclecloud.com <http://www.cyclecloud.com/>
>>
>>     http://twitter.com/cyclecomputing
>>
>>     On Thursday, 6 September, 2012 at 9:28 AM, Shrum, Donald C wrote:
>>
>>         Looks like the collector -
>>
>>         udp 0 0 10.178.6.5:41796 10.178.6.5:1980 ESTABLISHED
>>         580/condor_collecto
>>
>>         udp 0 0 10.178.6.5:43588 10.178.6.5:1980 ESTABLISHED
>>         580/condor_collecto
>>
>>         udp 0 0 10.178.6.5:48964 10.178.6.5:1980 ESTABLISHED
>>         580/condor_collecto
>>
>>         udp 0 0 10.178.6.5:40004 10.178.6.5:1980 ESTABLISHED
>>         580/condor_collecto
>>
>>         udp 0 0 10.178.6.5:47684 10.178.6.5:1980 ESTABLISHED
>>         580/condor_collecto
>>
>>         This was on the central manager. Next time I see it happen on
>>         a processing node I'll check there as well.
>>
>>         -----Original Message-----
>>
>>         From: condor-users-bounces@xxxxxxxxxxx
>>         <mailto:condor-users-bounces@xxxxxxxxxxx>
>>         [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Brian
>>         Candler
>>
>>         Sent: Thursday, September 06, 2012 8:36 AM
>>
>>         To: Condor-Users Mail List
>>
>>         Subject: Re: [Condor-users] 7.8.2 / running out of ports for 
>> UDP
>>
>>         On Thu, Sep 06, 2012 at 12:27:46PM +0000, Shrum, Donald C wrote:
>>
>>             I'm running redhat 6.3 with condor 7.8.2
>>
>>             On a number of my servers, both processing and on the
>>             central manager;
>>
>>             I find condor holding open a massive number of UDP ports.
>>             So many that
>>
>>             it blocks any new connections and DNS lookups fail.
>>
>>             Is this happening for anyone else?
>>
>>         Can you say which particular condor process is holding open
>>         the ports?
>>
>>         netstat -naup
>>
>>         (as root) should show you the process name and pid for each
>>         socket.
>>
>>         _______________________________________________
>>
>>         Condor-users mailing list
>>
>>         To unsubscribe, send a message to
>>         condor-users-request@xxxxxxxxxxx
>>         <mailto:condor-users-request@xxxxxxxxxxx> with a
>>
>>         subject: Unsubscribe
>>
>>         You can also unsubscribe by visiting
>>
>>         https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>>         The archives can be found at:
>>
>>         https://lists.cs.wisc.edu/archive/condor-users/
>>
>>         _______________________________________________
>>
>>         Condor-users mailing list
>>
>>         To unsubscribe, send a message to
>>         condor-users-request@xxxxxxxxxxx
>>         <mailto:condor-users-request@xxxxxxxxxxx> with a
>>
>>         subject: Unsubscribe
>>
>>         You can also unsubscribe by visiting
>>
>>         https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>>         The archives can be found at:
>>
>>         https://lists.cs.wisc.edu/archive/condor-users/
>>
>>
>>
>>
>>     _______________________________________________
>>
>>     Condor-users mailing list
>>
>>     To unsubscribe, send a message tocondor-users-request@xxxxxxxxxxx  
>> <mailto:condor-users-request@xxxxxxxxxxx>  with a
>>
>>     subject: Unsubscribe
>>
>>     You can also unsubscribe by visiting
>>
>>     https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>>
>>
>>     The archives can be found at:
>>
>>     https://lists.cs.wisc.edu/archive/condor-users/
>>
>>
>>
>> _______________________________________________
>> Condor-users mailing list
>> To unsubscribe, send a message tocondor-users-request@xxxxxxxxxxx  
>> with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/condor-users/
>
>
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx 
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/