[v2.4] IPv6 Neighbor Discovery appears to be broken for interfaces using CONFIG_SLIP_TAP=n

[zkrx]

We have been running Zephyr 1.14 on our custom boards for a long time. We have a somewhat exotic setup using CONFIG_NET_ROUTING=y on a device that has an ethernet interface on one side and several SLIP interfaces (each SLIP iface using CONFIG_SLIP_TAP=n) on the other. All our devices use IPv6. We are slowly catching up with latest Zephyr versions. We have now reached 2.4 and we noticed that our networking broke. A first inquiry showed that the neighbor table on our Zephyr router is wrong/incomplete on 2.4.

I could pinpoint the issue to the following commit:
debc8d6

Which was part of the following PR:
#16201

Since CONFIG_SLIP_TAP=n doesn't send MAC addresses on the wire, net_pkt_lladdr_src(pkt)->len will be 0 for these incoming NA packets, and nbr_lladdr.len will be assigned the value of 0. This means that the neighbor table will be incomplete/incorrect.

Therefore, I believe that the above commit broke ICMPv6 Neighbor Discovery for interfaces with CONFIG_SLIP_TAP=n. This problem is potentially made apparent by the fact that we also use CONFIG_NET_ROUTING=y. I guess that packets have trouble getting routed to the right interface if the neighbor table is incorrect.

Reverting that commit solves our problem. I think that this issue is still present on latest Zephyr since the offending line is still there, but I cannot test it since we are still too far behind.

[nordicjm]

We have now reached 2.4

It would be best that you try this specific feature in main rather than report a bug with a version of zephyr that is over 1.5 years old at this point and completely unsupported where development has long since continued on main and 2.7 (for LTS).

[rlubos]

I briefly checked the code and it seems that the issue will still be present on the current main.

To be honest I've got a bit confused about how this Target Link-Layer Address Neighbour Advertisement option is handled. The option carries the length field, so it seemed natural that the actual LL length should be read from the packet directly, however it turns out that the length field is in a unit of 8 octets:

8-bit unsigned integer. The length of the option
(including the type and length fields) in units of
8 octets. The value 0 is invalid.
Therefore it cannot be used reliably as for example an extended LL address for 802.15.4 would be padded and encoded at 16 bytes.

Because of this, I think we should stick to the current approach to determine LL address length, but perhaps introduce a fallback mechanism - if there's no LL address on a packet (as in the case of a dummy L2), get the LL address length from the interface, as it used to be done before.

@jukkar @alexanderwachter Any opinion on this?

[jukkar]

I am not sure what is the actual issue here. For TUN devices (that do not have L2), the link layer address is not used and for example IPv6 NAs are not needed for anything. For those kind of device, only IP routing can be used.

[zkrx]

Without diving too much into it, I can say the following:

Our dummy SLIP interfaces do have a MAC address as shown by the "net iface" shell command. See https://elixir.bootlin.com/zephyr/latest/source/drivers/net/slip.c#L421 (we changed the random generation bit to have reproducible and fixed MAC addresses, but that doesn't matter):

Interface 0x200344e0 (Dummy) [1]                                                                                                                                                                                                                                                                                                                                                               
================================                                                               
Link addr : EE:C1:78:A8:7B:60                                                                  
MTU       : 576                                                                                                                                                                                
IPv6 unicast addresses (max 2):                                                                
        fe80::ecc1:78ff:fea8:7b60 autoconf preferred infinite                                  
        fd11:8963:41:9:ecc1:78ff:fea8:7b60 autoconf preferred infinite                         
IPv6 multicast addresses (max 4):                                                              
        <none>                                                                                 
IPv6 prefixes (max 2):                                                                         
        <none>                                                                                 
IPv6 default router :                                                                                                                                                                          
        fe80::44f:f9ff:fe39:f54a                                                               
IPv6 hop limit           : 64                                                                  
IPv6 base reachable time : 30000                                                               
IPv6 reachable time      : 17675                                                               
IPv6 retransmit timer    : 0

When all is fine, these are actually resolved correctly using the ND protocol. ICMPv6 and TCP traffic is able to work just fine.

With the change from debc8d6, I can see that the neighbor table gets incoherent on our Zephyr router. I also have occasional <err> net_route: Src ll and Dst ll are same messages. Routing sometimes breaks completely, sometimes works for a minute or two. It is a bit random.

In both examples that follow, I am routing packets between EE:C1:78:A8:7B:60 (Zephyr dev #1) and DC:A6:32:00:61:A4 (Linux RPi). We effectively have the following architecture:

            EE:C1:78:A8:7B:60  SLIP (UART)                         Ethernet    DC:A6:32:00:61:A4
               Zephyr dev #1  <-------->  Zephyr dev #4 (router) <------------>   Linux RPi
                               SLIP (UART)  ^  ^
Zephyr dev #2 (not connected) <-------------+  |
                               SLIP (UART)     |
Zephyr dev #3 (not connected) <----------------+

Here is the neighbor table on Zephyr dev #4 (router) when things are fine (without debc8d6):

uart:~$ net nbr                                                                                                                                                                                
     Neighbor   Interface        Flags State     Remain  Link              Address                                                                                                             
[ 1] 0x2002fcd0 0x2003b960     0/1/1/0 reachable  22628  EE:C1:78:A8:7B:60 fe80::ecc1:78ff:fea8:7b60                                                                                           
[ 2] 0x2002fd24 0x2003b960     0/1/1/0 reachable  27168  EE:C1:78:A8:7B:60 fd11:8963:41:9:ecc1:78ff:fea8:7b60                                                                                  
[ 3] 0x2002fc28 0x2003bba0     0/1/0/0 stale          0  DC:A6:32:00:61:A4 fe80::dea6:32ff:fe00:61a4                                                                                           
[ 4] 0x2002fc7c 0x2003bba0     0/1/1/0 reachable   7167  DC:A6:32:00:61:A4 fd11:8963:2111::1

Here is an example of the neighbor table when things go south (with debc8d6):

uart:~$ net nbr
     Neighbor   Interface        Flags State     Remain  Link              Address
[ 1] 0x20000cf8 0x2000cc54     0/1/1/0 reachable  11794  DC:A6:32:00:61:A4 fe80::ecc1:78ff:fea8:7b60
[ 2] 0x20000d4c 0x2000cc54     0/1/1/0 reachable  12303  DC:A6:32:00:61:A4 fd11:8963:41:9:ecc1:78ff:fea8:7b60
[ 3] 0x20000ca4 0x2000ce58     0/1/1/0 reachable  17237  DC:A6:32:00:61:A4 fd11:8963:2111::1

In the second example, fd11:8963:41:9:ecc1:78ff:fea8:7b60 and fe80::ecc1:78ff:fea8:7b60 are assigned the wrong LL address (DC:A6:32:00:61:A4) and routing falls apart.

[jukkar]

You seem to have enabled TAP interface in Linux (as I see Ethernet mentioned in the description). If this is really the case, then you also need to enable CONFIG_SLIP_TAP=y in Zephyr side.

[rlubos]

I now wonder, if you rely on LL adressing in your router device, why using CONFIG_SLIP_TAP=n instead of CONFIG_SLIP_TAP=y which includes Ethernet L2? @jukkar got a point that pure SLIP does not support a concept of LL addressing. Even if we fix the NA handling as proposed above (which after reconsideration sounds more like a workaround than actual fix), you may sooner or later run into other issues.

[carlescufi]

@zkrx could you please respond to the comments by @jukkar and @rlubos?

[zkrx]

Hey, of course. I am still investigating this on my end, hence the delay.

To clarify my situation a bit, our "router" has a standard ethernet interface (hence "TAP") connected to a Linux device on one end, and multiple SLIP interfaces ("TUN") on the other. The Linux device estabilishes TCP connections with our SLIP devices through our zephyr router.

I tried playing with CONFIG_SLIP_TAP=y and it appears to solve the issue. The neighbor table looks fine and stable. The issue we have with this internally is that it will break backward compatibility with our older devices. There was no strong reason to choose CONFIG_SLIP_TAP=n years ago but now we are somewhat stuck with it. I guess that was initially done to shave a few bytes off our UART links which are half-duplex and prone to collisions.

Keeping CONFIG_SLIP_TAP=n, I'd like to try another approach which is adding routes (e.g. with net_route_add() ?). I'd like to redirect all packets arriving on ethernet interface A for destination prefix X to SLIP interface B (which shares that prefix). Interface B has only one device attached to it at a given time, but the device (and its IP address) can change (within the same prefix). I'm not sure yet if/how this is feasible. In particular, I don't understand what nexthop parameter to net_route_add() I should use given that there isn't any. Translated into Linux parlance, what I want to do in Zephyr is:
ip -6 route add fd11:8963:41:9::/64 dev slip0

As a side note, in case @rlubos ' workaround is not implemented, I now wonder if NDP frames should even be forwarded to non-TAP interfaces at all. Perhaps no device from those non-TAP interfaces should even appear in the neighbor table. This would then be clear that NDP is not supported for TAP=n.

Thank you all for your answers so far.

[rlubos]

My say here is that we should disable ND for non-TAP slip interfaces. I'm currently working on a feature which would allow to do that on per-interface basis, instead of disabling ND globally with Kconfig.

In particular, I don't understand what nexthop parameter to net_route_add() I should use given that there isn't any.

That's another reason the design seems to be flawed for non-TAP interfaces. The purpose of nexthop is to provide the LL address for the routed packet, but given that there's no LL addressing on non-TAP interfaces it seems to be error prone.

Anyway, I see that there is some fallback mechanism implemented when there is no entry in the routing table - the IPv6 stack will scan over all available interfaces to see if there's a prefix match, and if so, it'd forward the packet into that interface. Perhaps that's the way to go here.

[carlescufi]

Given that this does not seem to be an issue in the codebase itself, but rather an "abuse" of the implementation that stopped working, I am converting this issue to a Discussion. The conversation can continue there all the same.

[zkrx]

the IPv6 stack will scan over all available interfaces to see if there's a prefix match, and if so, it'd forward the packet into that interface.

So I had a look at the code in question [1]. My understanding, based on your message and my own testing, is that this fallback code will not be hit without disabling NDP. Which I cannot afford to do for all interfaces.

I'm currently working on a feature which would allow to do that on per-interface basis, instead of disabling ND globally with Kconfig.

If my above understanding is right, then this would be a good solution for us since I could disable NDP for our SLIP interfaces only and let it fallback on a prefix match. Would you have any kind of ETA? Or a ticket/discussion that I could subscribe to?

[1] https://github.com/zephyrproject-rtos/zephyr/blob/main/subsys/net/ip/net_if.c#L2407

[rlubos]

@zkrx I have some changes that allow to disable ND/MLD on specific interfaces here: https://github.com/rlubos/zephyr/commits/net/nd-mld-flags. however, I'm waiting for some feedback internally before I open a PR.