BT Infinity IPv6, Got There In The End

[sgucukoglu]

Hi all,

First of all, I apologise in advance if this post is loaded with jargon people can't understand. Feel free to move on; this post is for geeks who simply must have IPv6 support no matter what, yesterday. BT can (and should, dammit!) be deploying IPv6 for everybody through its gateways, leaving you with nothing to do. But if you want to be an early starter, feel free to follow along. This post is FYI. One of the support people recommended I come here to post for those people it might help, and I think it'll probably help somebody. I ask a couple of questions but basically IPv6 is now working, and I am once again a full participant on the open Internet. Yay!

I moved from cable, with a 1500-byte MTU. I use an Apple Time Capsule (same as Airport Extreme), which supports PPPoE, and plug it directly into the OpenReach modem; a nice fit. I use tunnelbroker.net to get IPv6 service. In theory, all I'd have to do is change the WAN interface from Ethernet to PPPoE and tell TunnelBroker.net about my new IPv4 address and everything would just work, right? Nope! At first I thought the loss of IPv6 was Apple's fault, for not setting an appropriate MTU on the upstream interface and thus borking the tunnel. Packets would leave but never return, i.e., TCP connections would happen but then hang; classic MTU misconfiguration. So I fired off a nastygram to Apple's mailing list, and was shot down by an Apple engineer. Didn't I know about the virtues of Path MTU Discovery? If they were going to add yet another value to their base station software, it had better be justified, and there is no good reason why a properly functioning network should require the user to configure anything. The ISP would negotiate the MTU, the LAN would operate at full MTU, and ICMP messages are used to bottleneck the flow to the PMTU. Yes, I get it - and they're right, of course. I've opened a bug report asking them to consider an option nevertheless, in case BT does not change its network (which it should). But if they were right, something else had to be wrong ...

Of course I know PPPoE has lower-than-1500-byte MTU (but see below). My tunnel provider would be assuming that I had a 1500-byte MTU, though, and that meant the only way I could get it to work is to reduce each NIC's MTU, so the advertised TCP MSS value would always be low enough for packets to fit into the network and I could at least browse the web. But hang on a minute - how was IPv4 working then? Surely, if PPP hadn't negotiated the PMTU, where were we learning the MTU?

http://netalyzr.icsi.berkeley.edu/restore/id=43ca253f-27707-cdc38e65-3092-4752-8bf7

The answer, of course, is that we weren't, but this took a bit of figuring out. I sent packets in either direction using ping with varying sizes and having the DF bit set. The maximum send size was 1492 bytes IPv4 payload, and the maximum receive size 1496. After that, packets got lost. The Netalyzr (great service!) report confirms it. There are a few things in here we can ignore which pertain to my defunct DNS software, but the two most interesting details are:

1. Packets which require fragmentation in the network do not reach us. This is really astoundingly bad, but doesn't directly pertain to IPv6.

2. Packets which are explicitly marked as Don't Fragment but which are too big to fit into the network just get lost, instead of having the ICMP "Packet too large, but DF bit set" message returned. This is astoundingly bad, but *does* pertain to IPv6 tunnels, effectively preventing anybody from running one. Notice the IP addresses of the BT routers between which these losses occur.

Well, yes, but I'm still browsing the interwebs over IPv4. How can we explain that? Tcpdump to the rescue. When my packets leave my network, their TCP MSS option value is *re-ajusted* to 1400, and the same is true in the return direction. This is *very, very bad*. It's basically a kick to the head of any non-TCP protocols, and means most applications and services *seem* to work, most of the time, but will break in subtle ways when large datagrams are involved. I know this is true because I run both machines involved in the packet capture.

Really, there was only one option left: contact Hurricane Electric (tunnelbroker.net) and get them to likewise reduce my IPv6 payload MTU to 1476. This they did, being such great guys, and I was at last able to use IPv6 tunnels without modification. It's a temporary workaround and could stop working at any moment, and it explicitly relies on my Apple base station clamping the MTU in the upstream direction to the minimum IPv6 MTU of 1280 bytes, lucky stars.

So there you have it. How to get, and how not to get, IPv6 working. 🙂

Now, for the questions:

1. In all tests I only used Apple's equipment for the MSS clamping check. I'd appreciate it if somebody could confirm it's happening on BT's network, just in case it's happening locally and I can do something about it.

2. The MTU. I understand these OpenReach thingies support baby jumbograms. That ought to mean that I can achieve the full 1500-byte MTU. Since it's clear that I've negotiated it, could it be the case that the Ethernet layer is dropping out before the IP routing infrastructure has a chance to generate the ICMP messages? This too wants testing by somebody with another gateway on hand, with the ping command to generate 1493+-byte payloads, with a PPP-negotiated MTU, if any. You can also run the test from another, remote machine to confirm that it isn't your own router generating the ICMP messages, with payloads of 1497+.

And that's it! Hope somebody, somewhere, has come away with information that'll help them get on IPv6. And BT, please get on with the deployment of IPv6, and unbreak your network.

Cheers,

Sabahattin

Edit: Plain text edit compressed white space on post.  God I hate forums!

Edit: Remove one final compressed whitespace, add missing word.  Wish I was in my newsreader.

[sgucukoglu]

Google has caught on to this post, so I'd better update it with some more information.

1.  HE are no longer responsive, which means I no longer have IPv6.  Their reasons may not need explanation; asking them to lower the MTU on a temporary, per-tunnel basis is a clean hack, but it's still a hack.

2.  BT are not rewriting MSS in the net; that is Apple, as suspected.  They use a value of 1400 on any PPPoE link to work around precisely this kind of issue.  BT's hubs do it too; it's controlled by the value of the "Upstream MTU" parameter.

3.  Yes, it's the physical link.  A linux box running rp-pppoe was able to receive 1500-byte payload (1508 including PPPoE).  The modem supports baby jumbos/baby giants, then.  Alas, BT's PPP infrastructure is buggered; it only negotiates max. 1492, and does not follow  RFC 4638 (tut, tut) but instead just forwards large frames on anyway.  Worse, they don't let the customer choose which they have maximum support for, which causes drop-outs for large packets (see above) when the Ethernet hardware doesn't support it.

4.  The gall of it all.  I'm trying to reach BT Wholesale to discuss some of these points (wish me luck ...).  Meantime, I'll just have to use my Cisco 881W to fiddle the link MTU for IPv6 and/or MSS readjustment, if baby jumbos are not available.  But what really annoys me is how much effort was required to work it all out.  For example, it is a common expectation that MSS clamping be done even if it breaks non-TCP protocols (see above).  And it is common for PPPoE implementations not to support baby jumbos even though the Ethernet hardware works, transparently supporting both failure cases.  It's a gigantic charlie foxtrot all right.

5.  The case against PPPoE.  PPPoE is an antiquated, legacy protocol for an antiquated, legacy mode of service provision.  The only good thing about PPPoE is that it runs PPP, which allows the network provider to have no awareness of the customer edge equipment and therefore to allow the customer and service provider full control over the services provided, like IPv6 and internal routing,  independent of the path between them.  But this is no consolation for a customer for whom PPPoE presents real difficulties in a faulty Internet full of PMTUD black holes (and especially those caused by one's own service provider assuming that everybody is next-generation and has 1500-byte payload MTU), is selfish in placing the burden upon the customer instead of the provider for layer-2 discrimination of traffic for accounting (as done with cable), and does not address the underlying problems that the infrastructure should support newer protocols (read: IPv6) and optimal routing from end to end.  Kill it already.

Cheers,

Sabahattin

[deasmi]

A quick note on HE.

They now have the option, on a per tunnel basis, to set the MTU within their admin interface, that helped me get a stable tunnel working.