Mailing List Archives

Authenticated access

 		[Computer Systems Lab]

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [Condor-users] 7.8.2 / running out of ports for UDP

Just a guess, but are you running with NO_DNS?

https://condor-wiki.cs.wisc.edu/index.cgi/tktview?tn=2691,3

On 09/06/2012 06:09 PM, Dan Bradley wrote:


Is there anything interesting in MasterLog?

It may be helpful to strace condor_master.

strace -p <insert-pid-of-master> -o master.strace

You can kill that after it has run for long enough to observe lots of
sockets being opened.

What is using port 1980?  Have you configured your collector to use that
port?

--Dan

On 9/6/12 11:02 AM, Shrum, Donald C wrote:


Hi Dan,

The problem persists so flocking was not related.  The condor master
is back up to 28,000+ open UDP ports.

Right now I’m just doing a periodic restart

[root@condor ~]# condor_status -master -format "%d\n"
MonitorSelfRegisteredSocketCount

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

connect: Resource temporarily unavailable

[root@condor ~]# service condor restart

Shutting down Condor (fast-shutdown mode)...  done.

Starting up Condor...    done.

[root@condor ~]# condor_status -master -format "%d\n"
MonitorSelfRegisteredSocketCount

1

Thanks for the help –

Don

FSU HPC

*From:*condor-users-bounces@xxxxxxxxxxx
[mailto:condor-users-bounces@xxxxxxxxxxx] *On Behalf Of *Dan Bradley
*Sent:* Thursday, September 06, 2012 10:53 AM
*To:* condor-users@xxxxxxxxxxx
*Subject:* Re: [Condor-users] 7.8.2 / running out of ports for UDP

Donald,

If you observe this problem again, see what the daemon is reporting in
its ClassAd:

condor_status -master -format "%d\n" MonitorSelfRegisteredSocketCount
<insert-hostname-here>

What's using port 1980?  The collector?

--Dan

On 9/6/12 9:28 AM, Shrum, Donald C wrote:

    As always, thanks Ian.

    We had flocking set up with another University and using the ‘this
    was one of the last things I touched’ trouble shooting method I
    just disabled flocking and Condor Connection Brokering (CCB_ADDRESS)

    That may have resolved the problem… we’ll see.

    --Don

    FSU HPC

    *From:*condor-users-bounces@xxxxxxxxxxx
    <mailto:condor-users-bounces@xxxxxxxxxxx>
    [mailto:condor-users-bounces@xxxxxxxxxxx] *On Behalf Of *Ian Chesal
    *Sent:* Thursday, September 06, 2012 9:34 AM
    *To:* Condor-Users Mail List
    *Subject:* Re: [Condor-users] 7.8.2 / running out of ports for UDP

    Donald,

    You could switch to TCP for collector updates:

    UPDATE_COLLECTOR_WITH_TCP = True

    See:
    http://research.cs.wisc.edu/condor/manual/v7.6/3_3Configuration.html#16701

    Or even better: switch to using the shared port daemon. This
    should help reduce the number of connections needed on any one
    machine. See:
    http://research.cs.wisc.edu/condor/manual/v7.6/3_7Networking_includes.html#32152

    Regards,

    - Ian

    --

    Ian Chesal

    Cycle Computing, LLC

    Leader in Open Compute Solutions for Clouds, Servers, and Desktops

    Enterprise Condor Support and Management Tools

    888.292.5320

    http://www.cyclecomputing.com <http://www.cyclecomputing.com/>

    http://www.cyclecloud.com <http://www.cyclecloud.com/>

    http://twitter.com/cyclecomputing

    On Thursday, 6 September, 2012 at 9:28 AM, Shrum, Donald C wrote:

        Looks like the collector -

        udp 0 0 10.178.6.5:41796 10.178.6.5:1980 ESTABLISHED
        580/condor_collecto

        udp 0 0 10.178.6.5:43588 10.178.6.5:1980 ESTABLISHED
        580/condor_collecto

        udp 0 0 10.178.6.5:48964 10.178.6.5:1980 ESTABLISHED
        580/condor_collecto

        udp 0 0 10.178.6.5:40004 10.178.6.5:1980 ESTABLISHED
        580/condor_collecto

        udp 0 0 10.178.6.5:47684 10.178.6.5:1980 ESTABLISHED
        580/condor_collecto

        This was on the central manager. Next time I see it happen on
        a processing node I'll check there as well.

        -----Original Message-----

        From: condor-users-bounces@xxxxxxxxxxx
        <mailto:condor-users-bounces@xxxxxxxxxxx>
        [mailto:condor-users-bounces@xxxxxxxxxxx] On Behalf Of Brian
        Candler

        Sent: Thursday, September 06, 2012 8:36 AM

        To: Condor-Users Mail List

        Subject: Re: [Condor-users] 7.8.2 / running out of ports for UDP

        On Thu, Sep 06, 2012 at 12:27:46PM +0000, Shrum, Donald C wrote:

            I'm running redhat 6.3 with condor 7.8.2

            On a number of my servers, both processing and on the
            central manager;

            I find condor holding open a massive number of UDP ports.
            So many that

            it blocks any new connections and DNS lookups fail.

            Is this happening for anyone else?

        Can you say which particular condor process is holding open
        the ports?

        netstat -naup

        (as root) should show you the process name and pid for each
        socket.

        _______________________________________________

        Condor-users mailing list

        To unsubscribe, send a message to
        condor-users-request@xxxxxxxxxxx
        <mailto:condor-users-request@xxxxxxxxxxx> with a

        subject: Unsubscribe

        You can also unsubscribe by visiting

        https://lists.cs.wisc.edu/mailman/listinfo/condor-users

        The archives can be found at:

        https://lists.cs.wisc.edu/archive/condor-users/

        _______________________________________________

        Condor-users mailing list

        To unsubscribe, send a message to
        condor-users-request@xxxxxxxxxxx
        <mailto:condor-users-request@xxxxxxxxxxx> with a

        subject: Unsubscribe

        You can also unsubscribe by visiting

        https://lists.cs.wisc.edu/mailman/listinfo/condor-users

        The archives can be found at:

        https://lists.cs.wisc.edu/archive/condor-users/




    _______________________________________________

    Condor-users mailing list

    To unsubscribe, send a message tocondor-users-request@xxxxxxxxxxx  <mailto:condor-users-request@xxxxxxxxxxx>  with a

    subject: Unsubscribe

    You can also unsubscribe by visiting

    https://lists.cs.wisc.edu/mailman/listinfo/condor-users



    The archives can be found at:

    https://lists.cs.wisc.edu/archive/condor-users/



_______________________________________________
Condor-users mailing list
To unsubscribe, send a message tocondor-users-request@xxxxxxxxxxx  with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/




_______________________________________________
Condor-users mailing list
To unsubscribe, send a message to condor-users-request@xxxxxxxxxxx with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/condor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/condor-users/