Re: [HTCondor-users] daemons not using IPv4 on unusable IPv6 network

[Todd L Miller <tlmiller@xxxxxxxxxxx>]

> we had an unexpected dual stack test run yesterday when our central node 
> [collector+negotiator] started using IPv6 due to network 
> misconfiguration.

	So it looks like the shared port daemon picked up the IPv6 address 
and started advertising it.

> - The Negotiator started talking to local daemons on IPv6 even though 
> forward resolution failed [2].

	This warning is, generally, only informative.  However,

> 07/10/17 06:40:01 (pid:3203476) (D_ALWAYS) WARNING: forward resolution of
> fe80::3a63:bbff:fe3f:59b4 doesn't match fe80::3a63:bbff:fe3f:59b4!

	In addition, this particular warning (IPV6 address literal not 
matching itself) is spurious; we just fixed it for the 8.6.5 release:

https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=6338.

> 07/10/17 06:40:01 (pid:3203476) (D_ALWAYS) PERMISSION DENIED to
> condor_pool@xxxxxxxxx from host fe80::3a63:bbff:fe3f:59b4 for command 421
> (Reschedule), access level DAEMON: reason: DAEMON authorization policy
> contains no matching ALLOW entry for this request; identifiers used for this
> host: fe80::3a63:bbff:fe3f:59b4, hostname size = 0, original ip address =
> fe80::3a63:bbff:fe3f:59b4

	condor_pool@ is usually associated with using PASSWORD 
authentication.  I'll guess here that the problem is that the link-local 
address doesn't resolve to 'gridka.de', and your ALLOW list contains 
'condor_pool@xxxxxxxxx'.  If your ALLOW list contains 'condor_pool@*', let 
me know, because we just found a problem with '*' being protocol-specific:

https://htcondor-wiki.cs.wisc.edu/index.cgi/tktview?tn=6340

> The main question for us is why did these components try using IPv6 
> anyway?

	I'll have to look into this.  Could you check your logs and verify 
the sinful they were trying to contact was

<10.97.13.108:9618?addrs=10.97.13.108-9618+[2a00-1398-10a-610d-3a63-bbff-fe3f-7a08]-9618&noUDP>

?  Adding D_HOSTNAME do SCHEDD_DEBUG (or STARTD_DEBUG, etc, as 
appropriate) should give you a lot information about what HTCondor was 
trying to do.

	After reading the code, it looks like HTCondor prefers public 
addresses over private addresses, and PREFER_IPV4 only changes the 
relative order of the protocol within those categories.  There's another 
layer of preferences on top, where if the private IPv4 address is declared 
as a private network, and the source's private network name is the same as 
the target's, HTCondor will use the private network.  See 
PRIVATE_NETWORK_NAME and PRIVATE_NETWORK_INTERFACE in the manual.  I don't 
know off the top of my head if these will work without CCB; it may also be 
easier simply to disable IPv6 on machines where it's known not to work 
(usually meaning those that have only a link-local IPv6 address).

	Should the default for ENABLE_IPV6 be FALSE if only link-local 
IPv6 address are found?

> which seem to imply that IPv6 is not used at all unless IPv4 is not 
> working.

	Just to be clear, HTCondor doesn't know (or have any way to know) 
if a particular protocol is "working", only if an address of that protocol 
is present on any of the machine's interfaces.  If you want HTCondor to 
never try any IPv6 address, set ENABLE_IPV6 = FALSE.  Otherwise:

PREFER_IPV4: Metaconfiguration; the default value for the next four settings.

ADVERTISE_IPV4_FIRST: If true, HTCondor will advertise IPv4 in its address 
lists before IPV6, indicating that it prefers to be contacted over IPV4. 
Otherwise, HTCondor will advertise IPV6 first.

IGNORE_TARGET_PROTOCOL_PREFERENCE: If true, HTCondor will ignore
target daemon's protocol preference, as indicated above, and instead 
select which protocol to use based on its own preferences.  By default,
HTCondor will prefer IPv6.

PREFER_OUTBOUND_IPV4: If true, when HTCondor will prefer IPv4 when
IGNORE_TARGET_PROTOCOL_PREFERENCE (above) is true.  Also, if 
IGNORE_DNS_PROTOCOL_PREFERENCE (below) is true, HTCondor will try 
IPv4 addresses returned by DNS first.

IGNORE_DNS_PROTOCOL_PREFERENCE: If true, HTCondor will sort DNS replies
by protocol.  (DNS replies are normally sorted in order of the host's
contact preferences.)  By default, IPv6 addresses will be sorted first.

- ToddM