Force10 E300 vs. Juniper MX480

Fri Jul 18 14:52:35 UTC 2008

Keith O'neill wrote:
> Force 10 is fine. I do suggest he go with the dual cam cards over the regular cards. I am not sure what Chris is talking about but I have used Force 10 for a long time, E, C and S series and have found it very stable. It will do everything you want and then some. The E300 is a good bang for the buck. Sure Foundry might be cheaper but I hear more complaining about Foundry than any other platform.
> 
> Chris you want to share what issues you have seen with Force 10.
> 
> Keith
> 
> ----- Original Message -----
> From: "Chris Marlatt" <cmarlatt at rxsec.com>
> To: "Joe Abley" <jabley at ca.afilias.info>
> Cc: "nanog" <nanog at merit.edu>
> Sent: Friday, July 18, 2008 7:43:33 AM (GMT-0500) America/New_York
> Subject: Re: Force10 E300 vs. Juniper MX480
> 
> Joe Abley wrote:
>> Hi all,
>>
>> An acquaintance who runs an ISP with an M7i on its border is looking to 
>> upgrade, because the M7i is starting to creak from all the flesh-tone 
>> MPEGs his customers are sharing. (How times have changed. Back when I 
>> was chasing packets, it was flesh-tone JPEGs.)
>>
>> He's looking at the MX480 and the E300.
>>
>> The MX480 is attractive because the M7i has been stable as a rock, and 
>> he's familiar with JUNOS.
>>
>> The E300 is attractive because it's half the price of the MX480, and has 
>> the potential to hold layer-2 cards as well as layer-3 ports which makes 
>> the price per port much more reasonable than the MX480. But he has no 
>> experience with Force10 at any ISO layer higher than 2.
>>
>> He doesn't have any exotic requirements beyond OSPF, OSPFv3, BGP, IP and 
>> IPv6. There's no MPLS in the picture, for example. However, he's going 
>> to want four or five full tables plus a moderate load of peering routes 
>> in there. And maybe VRRP.
>>
>> Thoughts from people who have tried one or the other, or both? Or who 
>> have faced this kind of problem, and came up with a different answer?
>>
>> Feel free to send mail off-list; I can summarise if there is interest.
>>
>>
>> Joe
>>
> 
> I would avoid Force10 if at all possible. In the network I managed I've 
> had some fairly surprising stability problems with their S series 
> switches and feature problems (or lack there of) on their E series. 
> Things you kind of scratch your head at and wonder what they were 
> thinking. Juniper on the other hand is indeed a bit pricier but quite a 
> stable platform. If he has to look at alternatives I would suggest 
> Foundry, either the RX-8, MLX-8, or XMR-8000 (depending on requirements) 
> for comparable models to the MX480.
> 
> Regards,
> 
> 	Chris
> 
> 

Considering I just had another issue pop up sure - I'd be glad to at 
this point.

As provided to another member who contacted me off list:
==========================================================
The S series problems were the worst - customer facing issues. 
<--snip-->. The list is noted in SFTOS and FTOS. Our design required 
layer 3 code on the S50N which "caused" some of these errors to present 
themselves:

- SFTOS: Limit of 8 ACL's (total ACL line count). Secondary assignments 
on the switch were "unprotected".

- SFTOS: OSPF required a specific ACL to form an adjacency even with a 
"default allow".

- SFTOS: If an uplink went down with OSPF running (ECMP) when the link 
was brought back up the OSPF adjacency would only form half way but 
would add a route. A 50/50 chance of success was the result.

- SFTOS: A "Transient Parity Error" crashed one of the S50's in 
production. No known cause.

- FTOS: The switch would lock during certain ARP operations (i.e. port 
flap). A hard reboot was necessary to recover the switch. <--snip-->

- FTOS: Random reboots preceded by "Low memory" errors. Our design would 
not / could not have consumed all the switch memory.

- FTOS: An upgrade from SFTOS to FTOS changes all the SNMP interface 
indexes causing lots of internal software to no longer be able to poll 
switch ports or monitor accurately.

- FTOS: Hard lock of the switch after an STP root change. The root 
change was not seen on any other switches (i.e. another bug in the S50 
code) and there were no events that should have caused a change in the 
topology.

The E series has more stable but like I said lacking some features. The 
most notable is the inability to do "normal" PBR. Pretty much any BGP 
attribute can't be used to build a policy. We were forced to dedicate 
vlans to certain policies as they could only match the traffic via an 
interface.

A minor annoyance is the timing for the management cpu causing ping 
times to look as though there is something wrong with the router. 
There's a paper out there somewhere explaining the cause for this and it 
has to do with the polling cycles of the board.

A snippet of a ping to a routing interface:
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=4 ttl=252 time=0.640 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=5 ttl=252 time=5.376 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=6 ttl=252 time=12.170 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=7 ttl=252 time=1.106 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=8 ttl=252 time=8.089 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=9 ttl=252 time=0.715 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=10 ttl=252 time=3.758 ms
64 bytes from xxx.xxx.xxx.xxx: icmp_seq=11 ttl=252 time=10.636 ms

The only other problem we've had with the E series is a BGP failure. The 
device failed over to its standby management module so the impact was 
limited. I don't hold that too much against them as I realize that no 
vendor is perfect. However the vast problems we've had with the S series 
and minor problems with the E bring into question the stability and 
unseen bugs with other software. <--snip-->

Hopefully the above is helpful. I'm sure my experience isn't unique or 
the norm. If everyone was having issues similar to mine they'd be out of 
business.
==========================================================

The most recent problem occuring today:
%FIB6-2-FIB6_HW_WRITE_ERROR: Failed to write entry into Host table.

Had to clear the fib in order to get communication with that host back up.

Of all the vendors I've worked with this is by _far_ the longest list of 
issues I've ever come across. I'm glad that you're having better success 
than I am. Believe me I wish I was in the same boat.

We've been using Foundry for a much longer period of time than we have 
Force10 and in comparison I personally no longer consider them 
comparable products.

Regards,

	Chris