[Ipv6-techsig] IPv6 PMTUD issues, some data

Tue Oct 6 17:24:43 NZDT 2009

Hi there,

I'm a fourth-year Waikato University student who works for WAND in his
spare time.

Earlier this year Geoff Huston wrote a very interesting article [1] in
which he describes and diagnoses a problem he had downloading an RFC
using IPv6. He determined that it was due to PMTUD failure (probably
caused by ICMPv6 filtering close to the web server) and suggests
lowering the MTU on interfaces used for IPv6 traffic to avoid this
problem. This and the resulting ipv6-techsig discussion [2] motivated me
to do a university project (supervised by Dr. Matthew Luckie) that
investigates problems with PMTUD in IPv6. It aims to determine the
prevalence of PMTUD failure in IPv6 by running tests to a large number
of IPv6-enabled web servers on the Internet.

As part of my investigation, I have written a module for scamper [3] (an
Internet measurement tool created by Matthew) that can be used to
determine whether PMTUD for a given web server is successful. It is
based on the the IPv4 PMTUD test used by tbit [4]. I won't bore you with
the exact details of the algorithm, but the general idea is:

Establish a connection to the web server and make an HTTP request for a
specific object. Do not acknowledge the response. Instead, send an
ICMPv6 Packet Too Big (PTB) message asking it to reduce the size of its
packets to 1280 bytes. If the web server retransmits the HTTP response
using a smaller packet size, conclude that PMTUD was successful.
Otherwise if the retransmitted response packet is not smaller than the
original, retransmit the PTB message. If after two retransmissions
(three PTB messages) the server still hasn't reduced the size of its
HTTP response packet, conclude that PMTUD failed.

The test halts if the original response packet is not larger than 1280
bytes (the MTU specified in the PTB message). I can't specify a MTU
smaller than this because most hosts will ignore an MTU smaller than
1280 bytes (the IPv6 Minimum Path MTU [5]).

A test to a particular host will result in one of:

RESULT_PMTUD_SUCCESS
The server successfully completed PMTUD. A smaller HTTP response packet
was received from the server after sending it a PTB message.

RESULT_PMTUD_FAIL
The server failed to complete PMTUD as we did not receive a smaller
response from it. The PTB messages are probably being filtered.

RESULT_RX_TOOSMALL
The response packet was too small to be able to continue the test. It
must be at least 1281 bytes in size.

RESULT_RX_NOACK
Did not receive an acknowledgement for the HTTP request.

RESULT_RX_NODATA
The HTTP request was acknowledged, but no HTTP response was received.

RESULT_TCP_RST
A TCP reset segment was received before the test could complete.

RESULT_TCP_ERROR
Unexpected packet received in response to the HTTP request.

RESULT_TCP_NOCONN
Failed to establish a TCP connection to the web server.

Test Setup:

I run the tests from a FreeBSD 7.2 box that sits in the lounge of my
flat. It has IPv6 connectivity using 6to4 and the tunnel interface on
our gateway box has an MTU of 1480 bytes.

Test Parameters:

An MSS of 1221 is advertised to avoid the effect of tunnels earlier in
the reverse path as much as possible. For reasons mentioned above, a
1280 byte MTU is specified in the PTB message.

Test Input:

The list of IPv6 web servers was derived from the Alexa Top 1 Million
Websites list [6]. Of these 1 million domains, only 627 had globally
routable IPv6 addresses that could be connected to on port 80 (sad eh?).
A script searched each of these for a URL to an object that was at least
1221 bytes in size. A request for such an object will hopefully result
in a large enough HTTP response packet.

Test Results:

Frequency of each result

  371 PMTUD_SUCCESS
  214 RX_TOOSMALL
   22 RX_NODATA
   13 TCP_NOCONN
    9 PMTUD_FAIL
    5 TCP_RST
    2 RX_NOACK
    1 TCP_ERROR

In addition to the outright failures (PMTUD_FAILs), we believe that the
results RX_NODATA and RX_NOACK also indicate PMTUD failure. They could
result when the server's HTTP response packets are discarded somewhere
along the path for being too big and the server never learns that this
is occurring (and neither do we). The server's response packets are
disappearing into a PMTU black hole.

A large number of hosts send HTTP response packets that are too small
(RX_TOOSMALL). Some of these could actually indicate PMTUD success in
the presence of 1280 byte tunnels. I will look into this further.

Frequency of advertised MSS values

  458 mss=1440
   60 mss=1220
   47 mss=1221 (the remote TCP echoing our advertised MSS)
   33 mss=1420
   13 mss=1380
   10 mss=0    (did not receive a SYN from the web server)
    2 mss=8940
    2 mss=1410
    2 mss=1300
    1 mss=65455
    1 mss=33160
    1 mss=1460
    1 mss=1436
    1 mss=1432
    1 mss=1416
    1 mss=1412
    1 mss=1374
    1 mss=1370
    1 mss=1340

The vast majority of hosts advertised a 1440 byte MSS. We suspect that
the 1220 byte MSSs are from hosts that have been explicitly configured
with 1280 byte MTUs and had their PMTUD disabled so as to avoid the ICMP
filtering problem.

My current university project is almost complete, but in the Summer I
would like to continue my investigation into IPv6 PMTUD. Can any of you
recommend another application layer protocol (preferably a simple one
like HTTP) that my PMTUD test could  support in the future? Bittorrent
could be a good one (what do you think Nathan?). Is there anything else
that you would be interested in me finding out?

Questions and comments are most welcome.

Cheers
Ben Stasiewicz

[1] http://cidr-report.org/ispcol/2009-01/mtu6.html

[2]
http://listserver.internetnz.net.nz/pipermail/ipv6-techsig/2009-February/000234.html

[3] http://www.wand.net.nz/scamper/

[4] http://www.icir.org/mallman/papers/tcp-evo-ccr05.ps

[5] http://www.ietf.org/rfc/rfc2460.txt

[6] http://s3.amazonaws.com/alexa-static/top-1m.csv.zip