Packet loss and inexplicable delays

Tue Dec 23 11:50:55 UTC 2003

Hello All,
   In the process of testing certain circuits delivered as parts of a WAN, 
we have found that two links are having serious packet loss and 
(inexplicable!) delays. To be more specific, they have packet loss on one 
direction, and huge delays on the other direction. The following diagram 
shows the situation (all equipment is made by Cisco):

1. Site A <-> Site B:

                            Delay
                         ----------->
   --------  ATM  --------  ATM  -------   E3   --------------
   | 7500 | <---> | 7200 | <---> | GSR | <----> | 6500/MSFC2 |
   --------       --------       -------        --------------
                         <-----------
                          Packet loss

The GSR connects to the E3 with an Engine0 12E3-SMB LC. The ATM link between 
the 7500 and the 7200 is shaped at 58 Mbps (ATM rate), while the link 
between the 7200 and the GSR is 155 Mbps. The E3 is an HDLC circuit. During 
the test, we noticed interface resets on the E3 link (line down). We thought 
that maybe the MSFC2 has prioritization problems (i.e. PPP keepalives are 
not prioritized over normal traffic), but that does not seem to be the case 
according to documentation found. LDP between the GSR and the 6500 was also 
dropped a few times (this is an MPLS Carrier-supporting-Carrier scheme).

2. Site A <-> Site C:

                          Packet loss
                         ----------->
   --------  ATM  --------  ATM  --------   E3   -------
   | 7500 | <---> | 7200 | <---> | 7200 | <----> | GSR |
   --------       --------       --------        -------
                         <-----------
                             Delay

The GSR connects to the E3 (HDLC) with an Engine0 6E3-SMB LC. The 
7500<->7200 ATM subinterface is shaped at 40 Mbps (which, if I am correct, 
does explain packet loss for small packets, but not for large --i.e. 1500 
byte-- packets due to cell tax). Interface/LPD resets did not happen during 
this experiment.

The packet loss is about 5% from end to end, and the delays vary (they grow 
up to 10 seconds!). The testing tool we are using is Rude/Crude, and the 
test PCs seem to handle the load in general. These test PCs are connected on 
the two end-routers through FastEthernet interfaces. The send-rate used for 
the two tests is 34 Mbps, in both directions. It is probably useful to 
mention that the delay is increasing as the packet size increases. The two 
GSRs are running IOS 12.0(26)S, which according to the caveats list does not 
seem to have any problems with HDLC E3 links under any circumstances.

Any ideas/insight at all, would be greatly welcome!

   Thank you in advance,
     Costas

-- 
Constantinos A. Kotsokalis || ckotso at grnet.gr
Greek Research and Technology Network || http://www.grnet.gr
56 Mesogeion Av., Ampelokipi, 11527 Athens, Greece
Tel: +30-210-7474249 || Cell: +30-697-7606620 || Fax: +30-210-7474490