Mailing List Archives

Authenticated access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] 7.8.2 / running out of ports for UDP

Just a general note to all and for documentation...  the problem went away after removing the NO_DNS =  true in condor_config.

Thanks for all the help.

-----Original Message-----
From: condor-users-bounces@xxxxxxxxxxx [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Rob de Graaf
Sent: Thursday, September 06, 2012 12:38 PM
To: Condor-Users Mail List
Subject: Re: [Condor-users] 7.8.2 / running out of ports for UDP

Try setting NETWORK_INTERFACE = <IP address>, that might fix it. Note that interface names won't work, at least they didn't when I ran into this.

https://lists.cs.wisc.edu/archive/condor-users/2012-June/msg00043.shtml

On 09/06/2012 06:31 PM, Shrum, Donald C wrote:
> Yes.  In fact I am running with no dns.
>
> -----Original Message-----
> From: condor-users-bounces@xxxxxxxxxxx 
> [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Rob de Graaf
> Sent: Thursday, September 06, 2012 12:26 PM
> To: Condor-Users Mail List
> Subject: Re: [Condor-users] 7.8.2 / running out of ports for UDP
>
> Just a guess, but are you running with NO_DNS?
>
> https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=2691,3
>
> On 09/06/2012 06:09 PM, Dan Bradley wrote:
>>
>> Is there anything interesting in MasterLog?
>>
>> It may be helpful to strace condor_master.
>>
>> strace -p <insert-pid-of-master> -o master.strace
>>
>> You can kill that after it has run for long enough to observe lots of 
>> sockets being opened.
>>
>> What is using port 1980?  Have you configured your collector to use 
>> that port?
>>
>> --Dan
>>
>> On 9/6/12 11:02 AM, Shrum, Donald C wrote:
>>>
>>> Hi Dan,
>>>
>>> The problem persists so flocking was not related.  The condor master 
>>> is back up to 28,000+ open UDP ports.
>>>
>>> Right now I'm just doing a periodic restart
>>>
>>> [root@condor ~]# condor_status -master -format "%d\n"
>>> MonitorSelfRegisteredSocketCount
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> connect: Resource temporarily unavailable
>>>
>>> [root@condor ~]# service condor restart
>>>
>>> Shutting down Condor (fast-shutdown mode)...  done.
>>>
>>> Starting up Condor...    done.
>>>
>>> [root@condor ~]# condor_status -master -format "%d\n"
>>> MonitorSelfRegisteredSocketCount
>>>
>>> 1
>>>
>>> Thanks for the help -
>>>
>>> Don
>>>
>>> FSU HPC
>>>
>>> *From:*condor-users-bounces@xxxxxxxxxxx
>>> [mailto:condor-users-bounces@xxxxxxxxxxx] *On Behalf Of *Dan Bradley
>>> *Sent:* Thursday, September 06, 2012 10:53 AM
>>> *To:* condor-users@xxxxxxxxxxx
>>> *Subject:* Re: [Condor-users] 7.8.2 / running out of ports for UDP
>>>
>>> Donald,
>>>
>>> If you observe this problem again, see what the daemon is reporting 
>>> in its ClassAd:
>>>
>>> condor_status -master -format "%d\n" 
>>> MonitorSelfRegisteredSocketCount <insert-hostname-here>
>>>
>>> What's using port 1980?  The collector?
>>>
>>> --Dan
>>>
>>> On 9/6/12 9:28 AM, Shrum, Donald C wrote:
>>>
>>>      As always, thanks Ian.
>>>
>>>      We had flocking set up with another University and using the 'this
>>>      was one of the last things I touched' trouble shooting method I
>>>      just disabled flocking and Condor Connection Brokering
>>> (CCB_ADDRESS)
>>>
>>>      That may have resolved the problem... we'll see.
>>>
>>>      --Don
>>>
>>>      FSU HPC
>>>
>>>      *From:*condor-users-bounces@xxxxxxxxxxx
>>>      <mailto:condor-users-bounces@xxxxxxxxxxx>
>>>      [mailto:condor-users-bounces@xxxxxxxxxxx] *On Behalf Of *Ian Chesal
>>>      *Sent:* Thursday, September 06, 2012 9:34 AM
>>>      *To:* Condor-Users Mail List
>>>      *Subject:* Re: [Condor-users] 7.8.2 / running out of ports for 
>>> UDP
>>>
>>>      Donald,
>>>
>>>      You could switch to TCP for collector updates:
>>>
>>>      UPDATE_COLLECTOR_WITH_TCP = True
>>>
>>>      See:
>>>
>>> http://research.cs.wisc.edu/condor/manual/v7.6/3_3Configuration.html
>>> #
>>> 16701
>>>
>>>      Or even better: switch to using the shared port daemon. This
>>>      should help reduce the number of connections needed on any one
>>>      machine. See:
>>>
>>> http://research.cs.wisc.edu/condor/manual/v7.6/3_7Networking_include
>>> s
>>> .html#32152
>>>
>>>      Regards,
>>>
>>>      - Ian
>>>
>>>      --
>>>
>>>      Ian Chesal
>>>
>>>      Cycle Computing, LLC
>>>
>>>      Leader in Open Compute Solutions for Clouds, Servers, and 
>>> Desktops
>>>
>>>      Enterprise Condor Support and Management Tools
>>>
>>>      888.292.5320
>>>
>>>      http://www.cyclecomputing.com <http://www.cyclecomputing.com/>
>>>
>>>      http://www.cyclecloud.com <http://www.cyclecloud.com/>
>>>
>>>      http://twitter.com/cyclecomputing
>>>
>>>      On Thursday, 6 September, 2012 at 9:28 AM, Shrum, Donald C wrote:
>>>
>>>          Looks like the collector -
>>>
>>>          udp 0 0 10.178.6.5:41796 10.178.6.5:1980 ESTABLISHED
>>>          580/condor_collecto
>>>
>>>          udp 0 0 10.178.6.5:43588 10.178.6.5:1980 ESTABLISHED
>>>          580/condor_collecto
>>>
>>>          udp 0 0 10.178.6.5:48964 10.178.6.5:1980 ESTABLISHED
>>>          580/condor_collecto
>>>
>>>          udp 0 0 10.178.6.5:40004 10.178.6.5:1980 ESTABLISHED
>>>          580/condor_collecto
>>>
>>>          udp 0 0 10.178.6.5:47684 10.178.6.5:1980 ESTABLISHED
>>>          580/condor_collecto
>>>
>>>          This was on the central manager. Next time I see it happen on
>>>          a processing node I'll check there as well.
>>>
>>>          -----Original Message-----
>>>
>>>          From: condor-users-bounces@xxxxxxxxxxx
>>>          <mailto:condor-users-bounces@xxxxxxxxxxx>
>>>          [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Brian
>>>          Candler
>>>
>>>          Sent: Thursday, September 06, 2012 8:36 AM
>>>
>>>          To: Condor-Users Mail List
>>>
>>>          Subject: Re: [Condor-users] 7.8.2 / running out of ports 
>>> for UDP
>>>
>>>          On Thu, Sep 06, 2012 at 12:27:46PM +0000, Shrum, Donald C wrote:
>>>
>>>              I'm running redhat 6.3 with condor 7.8.2
>>>
>>>              On a number of my servers, both processing and on the
>>>              central manager;
>>>
>>>              I find condor holding open a massive number of UDP ports.
>>>              So many that
>>>
>>>              it blocks any new connections and DNS lookups fail.
>>>
>>>              Is this happening for anyone else?
>>>
>>>          Can you say which particular condor process is holding open
>>>          the ports?
>>>
>>>          netstat -naup
>>>
>>>          (as root) should show you the process name and pid for each
>>>          socket.
>>>
>>>          _______________________________________________
>>>
>>>          Condor-users mailing list
>>>
>>>          To unsubscribe, send a message to
>>>          condor-users-request@xxxxxxxxxxx
>>>          <mailto:condor-users-request@xxxxxxxxxxx> with a
>>>
>>>          subject: Unsubscribe
>>>
>>>          You can also unsubscribe by visiting
>>>
>>>          https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>
>>>          The archives can be found at:
>>>
>>>          https://lists.cs.wisc.edu/archive/condor-users/
>>>
>>>          _______________________________________________
>>>
>>>          Condor-users mailing list
>>>
>>>          To unsubscribe, send a message to
>>>          condor-users-request@xxxxxxxxxxx
>>>          <mailto:condor-users-request@xxxxxxxxxxx> with a
>>>
>>>          subject: Unsubscribe
>>>
>>>          You can also unsubscribe by visiting
>>>
>>>          https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>
>>>          The archives can be found at:
>>>
>>>          https://lists.cs.wisc.edu/archive/condor-users/
>>>
>>>
>>>
>>>
>>>      _______________________________________________
>>>
>>>      Condor-users mailing list
>>>
>>>      To unsubscribe, send a message 
>>> tocondor-users-request@xxxxxxxxxxx
>>> <mailto:condor-users-request@xxxxxxxxxxx>  with a
>>>
>>>      subject: Unsubscribe
>>>
>>>      You can also unsubscribe by visiting
>>>
>>>      https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>
>>>
>>>
>>>      The archives can be found at:
>>>
>>>      https://lists.cs.wisc.edu/archive/condor-users/
>>>
>>>
>>>
>>> _______________________________________________
>>> Condor-users mailing list
>>> To unsubscribe, send a message tocondor-users-request@xxxxxxxxxxx
>>> with a
>>> subject: Unsubscribe
>>> You can also unsubscribe by visiting 
>>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>>
>>> The archives can be found at:
>>> https://lists.cs.wisc.edu/archive/condor-users/
>>
>>
>>
>> _______________________________________________
>> Condor-users mailing list
>> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx 
>> with a
>> subject: Unsubscribe
>> You can also unsubscribe by visiting
>> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>>
>> The archives can be found at:
>> https://lists.cs.wisc.edu/archive/condor-users/
>>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx 
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
>
> _______________________________________________
> Condor-users mailing list
> To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx 
> with a
> subject: Unsubscribe
> You can also unsubscribe by visiting
> https://lists.cs.wisc.edu/mailman/listinfo/condor-users
>
> The archives can be found at:
> https://lists.cs.wisc.edu/archive/condor-users/
>
_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/