Why Android says "Connected, no internet": a foreign IPv6 router advertisement from a 5G router (CPE)

How it all started

On my Android phone, Wi-Fi was connected, but it said "Connected, no internet". Everything else on the network worked fine (rebooting the phone changed nothing).

It all started when I was looking at the UniFi device bridge IoT (which is never in stock). I really wanted one: there is no wired network connection in my office, and sometimes I badly need one. Then I remembered my Raspberry Pi 3 (which had been gathering dust for a few years) and decided to build such a bridge myself on it, with a VXLAN tunnel to my MikroTik hEX. Then I decided that wasn't enough and remembered my ISP-provided 5G router, a Huawei CPE Pro 2 (from now on I'll just call it the CPE), which I wanted to use for receiving and sending SMS in emergencies (no internet needed for that). After all the devices were added to the network, this problem showed up on my phone.

Suspicion and verification

Because I had done a lot that day, at first I thought I had messed something up in the settings of the MikroTik hEX (it's also my DNS server) and DNS had broken. But that didn't explain why only one phone was affected. After that I started suspecting the CPE, because this wonder of Chinese industry had very few settings. I had also put it into LAN Only mode before connecting it, and all I could do there was set a static address and turn off the DHCP server, so at first I wasn't expecting any problems.

After I disconnected it from the network, the problems with the phone disappeared. After I connected it again, they came back. So our culprit was found. Now I had to figure out what it was doing to break the network for one Android phone and nobody else.

Android says "no internet" when its own connectivity check fails: it sends a request to its servers and waits for a response. So something in the network had changed that affected where the phone's requests went, or more precisely, that they now went nowhere.

Searching online, I found that it's not always enough to turn off DHCP, especially when the network has IPv6. That's how I came across router advertisements (RA) for IPv6: messages by which a router announces to everyone around that it is the gateway (yet another thing that was supposed to make admins' lives easier).

That was only an assumption, so it still needed to be verified. I found a few ways to do it.

The first, quick one: install the ndisc6 package on a Linux host on the same network and run rdisc6 -1 wlan0. It sends a request and shows the router's response. The downside is that the -1 flag shows only the first response.

The second, more reliable one: passive capture, which asks nothing and just listens.

sudo tcpdump -i wlan0 -e -nn -tttt -l -v 'icmp6 and ip6[40]==134' 

The -e flag shows the sender's MAC, so you can see right away who the RA came from.

What a router advertisement actually is

An RA is an ICMPv6 message (type 134) by which a router, without being asked, tells everyone in the segment: "I'm the gateway, here's the prefix to take addresses from, here's the DNS." It's partly the IPv6 analogue of DHCP, but it works differently: the client doesn't ask for an address, it builds one itself from the prefix. This mechanism is called SLAAC.

An RA is sent to the multicast address ff02::1 (all nodes in the segment) periodically, and also in response to a router solicitation, which a client sends right after it joins a network. That is exactly the request rdisc6 sends. By the standard (RFC 4861), the default maximum interval between RAs is 600 seconds and the minimum is roughly 200, so 5 seconds, as in my log, is very frequent.

What is inside:

• Router lifetime (seconds): how long the client considers the sender its default gateway. A value of 0 means "I'm no longer a router." Mine was 1800, i.e. 30 minutes.
• Router preference (low, medium, high): priority when there are several routers. The CPE had high.
• M and O flags: whether to get the address (M) and other configuration, such as DNS (O), through DHCPv6 instead of SLAAC. In my RA it was "Stateful address conf: No" and "Stateful other conf: Yes".
• Prefix information: the prefix itself (/64), the on-link flag (addresses from it are reachable directly in the segment), the autonomous flag (the client can build an address from the prefix on its own), as well as valid and preferred lifetimes.
• RDNSS: the address of a DNS server the client can use without DHCP.
• MTU and sender MAC (source link-layer address).

Two things make this dangerous in a situation like this:

1. There is no authentication. Any device in an L2 segment can send an RA, and clients trust everyone by default. Corporate switches have a protection for this, RA Guard (RFC 6105), which lets RAs through only from trusted ports, though not all of them have it. A home network usually has nothing like that. Without such protection, a forged RA lets an attacker replace the gateway and DNS, i.e. carry out a man-in-the-middle attack.
2. The client remembers the path. As long as the router lifetime hasn't expired, the client keeps the sender in its list of gateways, even if there are no new RAs. So after filtering, the old RA could keep working on the phone for up to 30 minutes, and reconnecting to Wi-Fi was supposed to clear it.

What was in the CPE's RA

The very first RA from the CPE explained everything. Here it is (MAC addresses and other addresses are masked):

Soliciting ff02::2 (ff02::2) on wlan0... Hop limit : 64 ( 0x40) Stateful address conf. : No Stateful other conf. : Yes Mobile home agent : No Router preference : high Neighbor discovery proxy : No Router lifetime : 1800 (0x00000708) seconds Reachable time : unspecified (0x00000000) Retransmit time : unspecified (0x00000000) Prefix : fdxx:xxxx:xxxx:xxxx::/64 On-link : Yes Autonomous address conf.: Yes Valid time : 7200 (0x00001c20) seconds Pref. time : 3600 (0x00000e10) seconds Recursive DNS server : fe80::xxxx:xxxx:xxxx:xxxx DNS server lifetime : 1200 (0x000004b0) seconds MTU : 1500 bytes (valid) Source link-layer address: XX:XX:XX:XX:XX:XX from fe80::xxxx:xxxx:xxxx:xxxx 

It contained three things that together produce the symptom:

• Router preference: high, router lifetime: 1800 seconds. The CPE announces itself as the default gateway, and with raised priority on top of that, so the client happily picks it over my UniFi.
• A prefix from the ULA range (fd00::/8), length /64. It is generated from the CPE's own MAC address. There is no internet access through this prefix, because I had disabled mobile data on purpose.
• RDNSS. The CPE hands itself out as an IPv6 DNS server too, even though it has nothing to answer with.

On top of that, RAs came every ~5 seconds, not once every few minutes. Over a few minutes of capture, grep on the log found 40 matches for this CPE's MAC.

How this left the phone without internet

After connecting to Wi-Fi, Android checks connectivity on its own: it sends a request to its servers and sees whether they respond. If not, it says "no internet" and, depending on settings, may even switch to mobile data.

In my case the chain looked like this: the network announces IPv6, and one of the "routers" with IPv6 in it is the CPE with high priority. The phone takes an address and DNS from it, the request goes nowhere, and the check fails. Other devices didn't notice, and iPhones didn't suffer from this problem. Most likely, Android watches more closely what arrives from the network, while other devices quietly switch to IPv4 when IPv6 doesn't work and don't show the user such a message.

Whether UniFi (my router to the internet) also sends RAs, I haven't checked either. In theory this can be found out from the same capture by filtering by UniFi's MAC address.

The fix: drop IPv6 from the CPE on the MikroTik hEX

The CPE had to stay on the network, because I still needed it. So I decided not to turn it off, but to drop only the IPv6 frames from its port. I connected the CPE to ether4 on the hEX (all traffic from this port goes through the router, so the filter can be placed right there) and added two bridge filter rules:

/interface bridge filter add chain=forward in-interface=ether4 mac-protocol=ipv6 action=drop add chain=input in-interface=ether4 mac-protocol=ipv6 action=drop 

The first rule drops the IPv6 that the CPE sends onward into the LAN (that is where the RAs go), the second the IPv6 addressed to the hEX itself.

At first it didn't work. The counters showed 1-2 packets, while RAs in the log kept coming every 5 seconds. The cause turned out to be hardware offload: ports ether2-5 in the bridge on the hEX have the H flag (hardware offload), i.e. the switch chip forwards multicast itself, and the bridge filter sees only a small part of the traffic. I turned offload off only on the port with the CPE:

/interface bridge port set [find interface=ether4] hw=no 

After that, the forward counter grew from 123 to 149 packets in a few seconds, and in the logs I no longer saw new RA entries. The filter doesn't touch IPv4, because the rule works only on mac-protocol=ipv6, so the CPE's web interface and the SMS API stayed available.

All that was left on the phone was to turn Wi-Fi off and on, because the old RA lives in memory until the router lifetime expires, i.e. up to 30 minutes. I did that, and the problem went away.

Limitations and alternatives

My solution works, but it isn't the only one, and it has a price. There are several options, and each suits a different situation:

1. Turn off IPv6 on the device itself. This is the cleanest, because it removes the source rather than the effect. The web interface of my CPE is very stripped down, and I couldn't find anything about IPv6 there.
2. A switch rule on the hEX. In theory it gives the same filter right in the chip, without turning offload off. Not every board supports this, and my hEX (E50UG) doesn't.
3. A separate VLAN for the device. Reliable, but it requires more changes in the network.

My variant has three limitations:

• A bridge filter works in software, so the port loses hardware acceleration. In my case, for a CPE that transfers almost nothing, it doesn't matter. For a fast device it would be noticeable.
• The filter is tied to the port. If you move the CPE to another port, the protection disappears and RAs go back into the network.
• The filter drops all IPv6 from the port. If you ever need IPv6 on it, this won't do.

Conclusions

When you add to your network a device that is itself a router (a mobile router, a VPN gateway, an access point in router mode), it is worth checking right away whether it sends out its own RAs and DHCP. rdisc6 or a short tcpdump will show this in a few seconds.

The symptom "the problem is only on Android" is also a useful hint. I think such cases happen more often than people write about them, it's just that they are rarely traced back to IPv6.