Strange public traceroutes return private RFC1918 addresses

Wed Feb 4 23:03:32 UTC 2004

On 3-feb-04, at 11:47, Michael.Dillon at radianz.com wrote:

>> Which (as discussed previously) breaks things like Path MTU Discovery,
>> traceroute,

> If RFC1918 addresses are used only on interfaces with jumbo MTUs
> on the order of 9000 bytes then it doesn't break PMTUD in a
> 1500 byte Ethernet world. And it doesn't break traceroute.

You mean if they use 9000 bytes + RFC 1918 for the internal links and 
1500 + real addresses for the external links there are no problems, 
even when people filter the RFC 1918 addresses? That would be correct 
in the case where this is a single organization network. But if it's a 
service provider network, there may be customers somewhere that connect 
over 1500 byte + links. (And never mind the fact that firewall admins 
are incredibly paranoid and also often filter RFC 1918 sources.)

> A more important question is what will happen as we move out
> of the 1500 byte Ethernet world into the jumbo gigE world.

Not as much as I'd hoped. My powerbook has gigabit ethernet but it's 
limited to 1500 byte frames.

> It's only a matter of time before end users will be running gigE
> networks and want to use jumbo MTUs on their Internet links.

The internet has always been a network with a variable MTU size. Even 
today under the iron rule of ether there are many systems with MTUs 
that aren't 1500. And yes, obviously people will want larger MTUs. I 
had the opportunity to work with a couple for boxes with 10 gigabit 
ethernet interfaces today. Unfortunately, I was unable to squeeze more 
than 1.5 gbit out of them over TCP. That's 125000 packets per second at 
1500 bytes, which makes no sense any which way you slice it. (And the 
driver did actually do 125k interrupts per second, which probably 
explains the poor throughput.)

> Could we all agree on a hierarchy of jumbo MTU sizes that
> with the largest sizes in the core and the smallest sizes at
> the edge? The increment in sizes should allow for a layer or
> two of encapsulation and peering routers should use the
> largest size MTU.

No need. Simply always use the largest possible MTU and make sure path 
MTU discovery works. If you have a range of maximum MTU sizes that is 
pretty close (9000 and 9216 are both common) it could make sense to 
standardize on the lowest in the range to avoid successive PMTUD drops 
but apart from that there is little to be gained by over-designing.

Oh yes: there were some calculations in other postings, which were 
quite misleading as they only looked at the 20 byte IP overhead. 
There's also TCP overhead (20 bytes), often a timestamp option (12 
bytes) and of course the ethernet overhead which is considerable: 8 
byte preamble, 14 byte header, 4 byte FCS and an inter frame gap that 
is equivalent to 12 bytes. So a 1500 byte IP packet takes up 1538 bytes 
on the wire while it only has a 1460 byte payload (94.9% efficiency). A 
9000 byte IP packet takes up 9038 bytes and delivers a 8960 byte 
payload (99.1%). 1520 bytes in a single packet would be 95% efficiency 
but fragmenting this packet would create a new IP packet with a 24 byte 
payload for a total of 44 bytes which is padded to 46 because of the 
ethernet minimum packet size, for a total bandwidth use on the wire of 
1618 bytes, making for an efficiency rating of 91.5%. (Fragmenting 1520 
in 1496 and 44 is pretty stupid by the way, 768 and 772 would be much 
better, thinking of the reasons why is left as an exercise for the 
reader.)