Port 80 Navigation Problem (was:FW: hey had eric sent you)

• Previous message (by thread): hey had eric sent you
• Next message (by thread): Telia v ATT?
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

My apologies to the list - I haven't slept in a day or two and
completely forgot to make a semi-intelligent subject line...
 
mjr

	-----Original Message-----
	From: Riley, Marty 
	Sent: Friday, March 12, 2004 22:18
	To: nanog at merit.edu
	Subject: FW: hey had eric sent you
	
	
	I'm running short on theories and options, so I thought I would
see if any other ISPs have seen this problem on your network(s).  If so,
what was the cure?
	 
	mjr
	 
	 
	-----Original Message-----
	

	The Unknown problem.

	 

	Symptoms: At random times dialup, dedicated, & internal network
users are unable to 

	          pass TCP traffic to off network sites.  ICMP and UDP
appears to be 

	          uneffected by the outage which lasts anywhere from 2
to 5 minutes.

	          

	          The problem appears to be wide spread with similar
reports from WVNET 

	          and other ISPS.  nTelos is experiencing a similar
problem but we have 

	          yet to confirm it is the same.

	          

	          Problem has changed in it's action but remained
similar enough to

	          consider it the same problem.

	 

	 

	Effected Platforms: Windows 2000 Pro, XP Home, XP Pro, & 2003
Server.

	 

	 

	Uneffected Platforms: Unix, MacOS (?)

	 

	 

	History: During the week of 2/9/04 the call center started
recieving reports of 

	         users being unable to connect to sites off the CityNet
network.  Sites 

	         hosted on the internal network are uneffected by the
outage.   

	 

	         Initally it was thought to be a Internet Explorer
problem possably caused

	         by the KB832894 / IE SP1 or other updates but after
further investigation 

	         it was found that Mozilla users were encountering the
same problem.  

	 

	         After several days of testing it was determined that
during the outage any 

	         TCP session started on any port would fail.  TCP
sessions started before 

	         the outage continue to work and show no ill effects
from the outage.

	   

	         After logging connection attempts at various intervals
on many machines

	         there appears to be no sort of pattern in the outages.
Most machines 

	         encounter the problem, some more than others and a few
do not encounter

	         it at all.  The duration and frequency of the outage is
very fluid.

	         

	         During an outage, we can verify that the packet does
seem to leave and reenter

	         the network:

	 

	Mar  5 22:28:04 pittpa-chaswv-ds3 17083: SLOT 2:6d20h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) ->
216.41.224.3(80), 1 packet

	Mar  5 22:28:09 pittpa-chaswv-ds3 17084: SLOT 1:6d20h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.14.174(3376), 1 packet

	Mar  5 22:28:09 pittpa-chaswv-ds3 17085: SLOT 2:6d20h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) ->
216.41.224.3(80), 1 packet

	Mar  5 22:28:09 pittpa-chaswv-ds3 17086: SLOT 1:6d20h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.14.174(3378), 1 packet

	Mar  5 22:33:24 pittpa-chaswv-ds3 17089: SLOT 1:6d20h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.14.174(3378), 7 packets

	Mar  5 22:33:24 pittpa-chaswv-ds3 17090: SLOT 1:6d20h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.14.174(3376), 17 packets

	Mar  5 22:33:58 pittpa-chaswv-ds3 17092: SLOT 2:6d20h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3378) ->
216.41.224.3(80), 7 packets

	Mar  5 22:33:58 pittpa-chaswv-ds3 17093: SLOT 2:6d20h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) ->
216.41.224.3(80), 18 packets

	 

	Mar  5 00:58:30 pittpa-clarwv-ds3 16062: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) ->
216.41.224.3(80), 1 packet

	Mar  5 00:58:30 pittpa-clarwv-ds3 16063: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3183), 1 packet

	Mar  5 01:03:28 pittpa-clarwv-ds3 16067: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) ->
216.41.224.3(80), 1 packet

	Mar  5 01:03:28 pittpa-clarwv-ds3 16068: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3217), 1 packet

	Mar  5 01:03:34 pittpa-clarwv-ds3 16069: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3228) ->
216.41.224.3(80), 1 packet

	Mar  5 01:03:34 pittpa-clarwv-ds3 16070: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3228), 1 packet

	Mar  5 01:03:39 pittpa-clarwv-ds3 16072: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3239) ->
216.41.224.3(80), 1 packet

	Mar  5 01:03:47 pittpa-clarwv-ds3 16073: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) ->
216.41.224.3(80), 74 packets

	Mar  5 01:04:13 pittpa-clarwv-ds3 16075: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3183), 72 packets

	Mar  5 01:08:46 pittpa-clarwv-ds3 16078: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3218) ->
216.41.224.3(80), 4 packets

	Mar  5 01:08:46 pittpa-clarwv-ds3 16079: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3217) ->
216.41.224.3(80), 3 packets

	Mar  5 01:08:46 pittpa-clarwv-ds3 16080: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3221) ->
216.41.224.3(80), 19 packets

	Mar  5 01:08:46 pittpa-clarwv-ds3 16081: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3228) ->
216.41.224.3(80), 5 packets

	Mar  5 01:08:46 pittpa-clarwv-ds3 16082: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3229) ->
216.41.224.3(80), 6 packets

	Mar  5 01:08:46 pittpa-clarwv-ds3 16083: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3236) ->
216.41.224.3(80), 9 packets

	Mar  5 01:08:47 pittpa-clarwv-ds3 16084: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3233) ->
216.41.224.3(80), 12 packets

	Mar  5 01:08:47 pittpa-clarwv-ds3 16085: SLOT 2:5d22h:
%SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3239) ->
216.41.224.3(80), 21 packets

	Mar  5 01:09:12 pittpa-clarwv-ds3 16087: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3239), 19 packets

	Mar  5 01:09:12 pittpa-clarwv-ds3 16088: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3228), 4 packets

	Mar  5 01:09:12 pittpa-clarwv-ds3 16089: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3217), 2 packets

	Mar  5 01:09:12 pittpa-clarwv-ds3 16091: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3218), 3 packets

	Mar  5 01:09:13 pittpa-clarwv-ds3 16092: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3221), 17 packets

	Mar  5 01:09:13 pittpa-clarwv-ds3 16093: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3229), 5 packets

	Mar  5 01:09:13 pittpa-clarwv-ds3 16094: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3236), 7 packets

	Mar  5 01:09:13 pittpa-clarwv-ds3 16096: SLOT 1:5d22h:
%SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
69.43.23.23(3233), 9 packets

	 

	         

	         Network analysis showed significant amounts of spoofed
multicast traffic and 

	         odd arp traffic.

	 10:17:16.416222 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp
278

	 10:17:16.421886 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp
334

	 10:17:16.423873 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp
262

	 10:17:16.426948 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp
254

	 10:17:16.432095 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp
298

	 10:17:16.435921 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp
274

	 10:17:16.439959 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp
328

	 10:17:16.445317 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp
326

	 10:17:16.449688 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp
330

	 10:17:16.463537 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp
322

	 

	         Steps were taken to elminiate the spoofed traffic on
the 

	         routers and access servers in the form of ACLs and
filter lists.

	             Neither have eliminiated the problem... but to what
extent they might

	             have helped has yet to be determined.

	 

	         The problem is still occuring, for some users the
duration of the outage 

	         seems to have shortened other users notice no
difference.  It is not yet

	         known if the filtering on the routers and access
servers or the conversion

	         to the 10.x.x.x network has made any difference.  We
should have a better idea

	         in the upcoming days.

	 

	What Doesn't help: Removing windows updates.  

	                   Turning off XP firewall.

	                   Searching for malware. (SpyBot-SD, Adaware)

	                   Virus scanning (Various softwares)

	                   Specifying dns servers.

	                   Reinstalling windows.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.nanog.org/pipermail/nanog/attachments/20040312/3fddeeba/attachment.html>

• Previous message (by thread): hey had eric sent you
• Next message (by thread): Telia v ATT?
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]