Juniper MX80 strange dst MAC address behavior

Tue Nov 7 21:01:49 UTC 2017

Today I was investigating strange unknown unicast traffic in LAN of IX 
which I operate. It was about 200 kbps of constant unknown unicast load.
Unknown unicast is a rare ocasion in IX LAN as participants MAC 
addresses are almost persistent.

I added a server in the vlan and started sniffing:
all the unicast was coming from one participant MAC. One of the frames:
XX:XX:XX:XX:ee:98 > 5e:5c:ab:31:c0:cb, ethertype IPv4 (0x0800), length 
1434: XX.XX.16.30.80 > XX.XX.76.137.41934: Flags [.], seq 
1619512599:1619513967, ack 3595347045, win 235, options [nop,nop,TS val 
3650267181 ecr 1841068329], length 1368: HTTP
Okay, destination MAC 5e:5c:ab:31:c0:cb isn't really in switches FDB 
table.

I looked up the routing table and figured out that router announcing the 
bestpath for XX.XX.76.137 has MAC 5e:5e:ab:31:c0:cb.

So customers sent a frame to: 5e:*5c*:ab:31:c0:cb
The right address should be : 5e:*5e*:ab:31:c0:cb

I tried next packet in dump, the same story:
customer sends to: *90*:c6:9a:*e5*:2f:c1
right mac is     : *b0*:c6:9a:*e4*:2f:c1

I converted differencing bytes to binary representation:
90: 10010000
b0: 10110000

e5: 11100101
e4: 11100100

I guess all the unknown unicast frames MAC addresses were having the 
same slightly difference from the right ones.

So the router in general works fine (it has several gigabits of traffic 
in the IX) but sometimes it changes one or two bits in frame's 
destination MAC address.
My guess it is caused by a large ARP table on customer's router. The 
router may have some tricky lookup algorithm preceding constant 
calculating speed over accuracy.
I called their NOC, they said it is Juniper MX80 and also confirmed that 
it has more than 4k ARPs.

Have anybody encountered with that kind of issue?