FW: hey had eric sent you

• Previous message (by thread): Load Balancing Multiple DS3s (outgoing) on a 7500
• Next message (by thread): FW: hey had eric sent you
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

I'm running short on theories and options, so I thought I would see if
any other ISPs have seen this problem on your network(s).  If so, what
was the cure?
 
mjr
 
 
-----Original Message-----


The Unknown problem.

 

Symptoms: At random times dialup, dedicated, & internal network users
are unable to 

          pass TCP traffic to off network sites.  ICMP and UDP appears
to be 

          uneffected by the outage which lasts anywhere from 2 to 5
minutes.

          

          The problem appears to be wide spread with similar reports
from WVNET 

          and other ISPS.  nTelos is experiencing a similar problem but
we have 

          yet to confirm it is the same.

          

          Problem has changed in it's action but remained similar enough
to

          consider it the same problem.

 

 

Effected Platforms: Windows 2000 Pro, XP Home, XP Pro, & 2003 Server.

 

 

Uneffected Platforms: Unix, MacOS (?)

 

 

History: During the week of 2/9/04 the call center started recieving
reports of 

         users being unable to connect to sites off the CityNet network.
Sites 

         hosted on the internal network are uneffected by the outage.   

 

         Initally it was thought to be a Internet Explorer problem
possably caused

         by the KB832894 / IE SP1 or other updates but after further
investigation 

         it was found that Mozilla users were encountering the same
problem.  

 

         After several days of testing it was determined that during the
outage any 

         TCP session started on any port would fail.  TCP sessions
started before 

         the outage continue to work and show no ill effects from the
outage.

   

         After logging connection attempts at various intervals on many
machines

         there appears to be no sort of pattern in the outages.  Most
machines 

         encounter the problem, some more than others and a few do not
encounter

         it at all.  The duration and frequency of the outage is very
fluid.

         

         During an outage, we can verify that the packet does seem to
leave and reenter

         the network:

 

Mar  5 22:28:04 pittpa-chaswv-ds3 17083: SLOT 2:6d20h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) ->
216.41.224.3(80), 1 packet

Mar  5 22:28:09 pittpa-chaswv-ds3 17084: SLOT 1:6d20h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.14.174(3376), 1 packet

Mar  5 22:28:09 pittpa-chaswv-ds3 17085: SLOT 2:6d20h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) ->
216.41.224.3(80), 1 packet

Mar  5 22:28:09 pittpa-chaswv-ds3 17086: SLOT 1:6d20h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.14.174(3378), 1 packet

Mar  5 22:33:24 pittpa-chaswv-ds3 17089: SLOT 1:6d20h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.14.174(3378), 7 packets

Mar  5 22:33:24 pittpa-chaswv-ds3 17090: SLOT 1:6d20h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.14.174(3376), 17 packets

Mar  5 22:33:58 pittpa-chaswv-ds3 17092: SLOT 2:6d20h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) ->
216.41.224.3(80), 7 packets

Mar  5 22:33:58 pittpa-chaswv-ds3 17093: SLOT 2:6d20h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) ->
216.41.224.3(80), 18 packets

 

Mar  5 00:58:30 pittpa-clarwv-ds3 16062: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) ->
216.41.224.3(80), 1 packet

Mar  5 00:58:30 pittpa-clarwv-ds3 16063: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3183), 1 packet

Mar  5 01:03:28 pittpa-clarwv-ds3 16067: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) ->
216.41.224.3(80), 1 packet

Mar  5 01:03:28 pittpa-clarwv-ds3 16068: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3217), 1 packet

Mar  5 01:03:34 pittpa-clarwv-ds3 16069: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3228) ->
216.41.224.3(80), 1 packet

Mar  5 01:03:34 pittpa-clarwv-ds3 16070: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3228), 1 packet

Mar  5 01:03:39 pittpa-clarwv-ds3 16072: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3239) ->
216.41.224.3(80), 1 packet

Mar  5 01:03:47 pittpa-clarwv-ds3 16073: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) ->
216.41.224.3(80), 74 packets

Mar  5 01:04:13 pittpa-clarwv-ds3 16075: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3183), 72 packets

Mar  5 01:08:46 pittpa-clarwv-ds3 16078: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3218) ->
216.41.224.3(80), 4 packets

Mar  5 01:08:46 pittpa-clarwv-ds3 16079: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) ->
216.41.224.3(80), 3 packets

Mar  5 01:08:46 pittpa-clarwv-ds3 16080: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3221) ->
216.41.224.3(80), 19 packets

Mar  5 01:08:46 pittpa-clarwv-ds3 16081: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3228) ->
216.41.224.3(80), 5 packets

Mar  5 01:08:46 pittpa-clarwv-ds3 16082: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3229) ->
216.41.224.3(80), 6 packets

Mar  5 01:08:46 pittpa-clarwv-ds3 16083: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3236) ->
216.41.224.3(80), 9 packets

Mar  5 01:08:47 pittpa-clarwv-ds3 16084: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3233) ->
216.41.224.3(80), 12 packets

Mar  5 01:08:47 pittpa-clarwv-ds3 16085: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3239) ->
216.41.224.3(80), 21 packets

Mar  5 01:09:12 pittpa-clarwv-ds3 16087: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3239), 19 packets

Mar  5 01:09:12 pittpa-clarwv-ds3 16088: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3228), 4 packets

Mar  5 01:09:12 pittpa-clarwv-ds3 16089: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3217), 2 packets

Mar  5 01:09:12 pittpa-clarwv-ds3 16091: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3218), 3 packets

Mar  5 01:09:13 pittpa-clarwv-ds3 16092: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3221), 17 packets

Mar  5 01:09:13 pittpa-clarwv-ds3 16093: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3229), 5 packets

Mar  5 01:09:13 pittpa-clarwv-ds3 16094: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3236), 7 packets

Mar  5 01:09:13 pittpa-clarwv-ds3 16096: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3233), 9 packets

 

         

         Network analysis showed significant amounts of spoofed
multicast traffic and 

         odd arp traffic.

 10:17:16.416222 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 278

 10:17:16.421886 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 334

 10:17:16.423873 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 262

 10:17:16.426948 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 254

 10:17:16.432095 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 298

 10:17:16.435921 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 274

 10:17:16.439959 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 328

 10:17:16.445317 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 326

 10:17:16.449688 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 330

 10:17:16.463537 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 322

 

         Steps were taken to elminiate the spoofed traffic on the 

         routers and access servers in the form of ACLs and filter
lists.

             Neither have eliminiated the problem... but to what extent
they might

             have helped has yet to be determined.

 

         The problem is still occuring, for some users the duration of
the outage 

         seems to have shortened other users notice no difference.  It
is not yet

         known if the filtering on the routers and access servers or the
conversion

         to the 10.x.x.x network has made any difference.  We should
have a better idea

         in the upcoming days.

 

What Doesn't help: Removing windows updates.  

                   Turning off XP firewall.

                   Searching for malware. (SpyBot-SD, Adaware)

                   Virus scanning (Various softwares)

                   Specifying dns servers.

                   Reinstalling windows.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20040312/185b3f17/attachment.html>

• Previous message (by thread): Load Balancing Multiple DS3s (outgoing) on a 7500
• Next message (by thread): FW: hey had eric sent you
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]