Re: [Condor-devel] localhost on linux

[Nick LeRoy <nleroy@xxxxxxxxxxx>]

On Tue September 11 2007, Douglas L Thain wrote:
> Howdy all -

Following up on this, as I've been tasked with solving it.

> As I'm sure everyone knows by now, Condor gets confused when
> Linux has an /etc/hosts with the following configuration:
>
>  	127.0.0.1 hedwig localhost localhost.localdomain
>
> Why does Condor get confused?  It attempts to determine
> the local IP address and full DNS name by the following procedure:
>
> 1 - Use uname to determine the short hostname. (hedwig)
> 2 - Use gethostbyname(shortname) to find the IP address. (129.74.162.69)
> 3 - Use gethostbyaddr(address) to find the full name. (hedwig.cse.nd.edu)
>
> The problem is, step 2 fails if /etc/hosts has the above configuration,
> because the IP address is found to be 127.0.0.1, and the full name
> is found to be localhost.localdomain.  Hilarity ensues.
>
> The solution (which is suggested in the masterlog) is to
> change /etc/hosts to be the following:
>
>  	127.0.0.1 localhost localhost.localdomain
>
> Now, if this was a rare problem, then that's probably enough said.
>
> But, it's not rare any more.  EVERY Linux installation of Condor
> that I have seen in the last three years has suffered from this
> problem, causing long debugging sessions, frustrated users,
> and people giving up because Condor "doesn't work".
>
> May I suggest that it would be a good PR move to find a better
> solution to this problem?
>
> Some ideas for discussion:
>
> 1 - Instead of logging the problem in an obscure file, modify the Condor
> tools to report an obnoxious error message that states your Linux
> installation is BROKEN, and here is how to fix it.

Comments:  *Very* easy to implement.

> 2 - Extract more local information.  If the local uname is not a full
> dns name, then infer the containing domain from /etc/resolv.conf,
> and try to resolve that combination of names.

I don't think that this is viable...  I don't think that we can rely 
on /etc/resolv.conf -- the host may not have a DNS record that matches it's 
view of it's own host name.  Also, I don't think that there's a API for 
walking through the file.  (I could be wrong on this point, however).


> 3 - Take the naming system out of the loop.  Make a TCP connection to
> the collector as if doing a condor_status, then a getsockname to
> get the local IP address with respect to the collector.

Not sure what I think about this one.  Like Greg's #4 below, this probably 
doesn't correctly handle the case in which the CM is also running a startd 
and/or schedd.

> Thoughts?  Ideas?

To these, we could add:

4 - (Greg T) Change BIND_ALL_INTERFACES to default as true.  This does solve 
the problem in many cases.


5 - Modify the code to walk through all available interfaces, and rank them 
based on the "publicness" of it's IP address.  127.* interfaces would get the 
lowest score, 192.168.* (and related addresses) would get a middle score, and 
truly public addresses would get the highest score.  Condor would then bind 
to the interface with the highest score.  For Linux, we can gather this info 
via the netdevice(7) call -- I don't know yet how we could gather this info 
on other platforms (other than, I suppose, running /sbin/ifconfig and parsing 
it's output).  This behavior could be turned off via a configuration knob 
(yeah, another one), perhaps something like BIND_BEST_INTERFACE = true/false.


#4 doesn't handle the case in which the machine in question is the central 
manager and is running a startd and/or schedd, the ads from these would have 
the address of the loopback (over which they would contact the collector).


Thoughts?

Also, I've now modified daemon core and Sock and a couple of other places to 
handle loopbacks other than 127.0.0.1 (it now masks it with 255.0.0.0 to 
check whether the given address is a loopback).  This is to handle systems 
like OpenSuse which now add another address of 127.0.0.2 -- in /etc/hosts, 
the .1 address is "localhost", and the .2 represents the host name 
(i.e. "127.0.0.2 myhost").  I haven't committed this code yet, but it does 
appear to work.

Thanks

-Nick


-- 
           <<< The answer is out there, Neo. >>>
 /`-_    Nicholas R. LeRoy               The Condor Project
{     }/ http://www.cs.wisc.edu/~nleroy  http://www.cs.wisc.edu/condor
 \    /  nleroy@xxxxxxxxxxx              The University of Wisconsin
 |_*_|   608-265-5761                    Department of Computer Sciences