Weird network problem.

• Previous message (by thread): Weird network problem.
• Next message (by thread): thanks for the great response on wholesalebandwidth.com major abuser
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]

Hey Bob

The problem we were seeing was in fact different than the one you seem
to have.  It was specific to IE and was corrected by a later MS patch. 
If it is any help, I can give you an account on the network to work with
in your diagnosis.  Send me an email off list or call.

I notice in your documentation that you mention a "conversion to the
10.x.x.x" network.  Are you NAT'ing?  

Herman Harless
Director Advanced Data Services
NTELOS, Inc.



On Fri, 2004-03-12 at 10:06, Bob Harden wrote:
> 
> Has anybody on the list ran into any problems like this?  We don't have
> a firewall and have tried with and without our ACL's but get the same
> results.  Below is our internal white paper on the situation.  If
> anybody has any ideas they would be much appreciated.
> 
> Thanks,
> 
> Bob Harden
> 
> 
> 
> 
> Network problem information.
>  
> Symptoms: At random times dialup, dedicated, & internal network users
> are unable to 
>           pass TCP traffic to off network sites.  ICMP and UDP appears
> to be 
>           uneffected by the outage which lasts anywhere from 2 to 5
> minutes.
>           
>           The problem appears to be wide spread with similar reports
> from WVNET 
>           and other ISPS.  nTelos is experiencing a similar problem but
> we have 
>           yet to confirm it is the same.
>         
> Effected Platforms: Windows 2000 Pro, XP Home, XP Pro, & 2003 Server.
>  
> 
> Uneffected Platforms: Unix & MacOS 
>  
> 
> History: During the week of 2/9/04 the call center started recieving
> reports of 
>          users being unable to connect to sites off the network.  Sites 
>          hosted on the internal network are uneffected by the outage.   
>  
>          Initally it was thought to be a Internet Explorer problem
> possably caused
>          by the KB832894 / IE SP1 or other updates but after further
> investigation 
>          it was found that Mozilla users were encountering the same
> problem.  
>  
>          After several days of testing it was determined that during the
> outage any 
>          TCP session started on any port would fail.  TCP sessions
> started before 
>          the outage continue to work and show no ill effects from the
> outage.
>    
>          After logging connection attempts at various intervals on many
> machines
>          there appears to be no sort of pattern in the outages.  Most
> machines 
>          encounter the problem, some more than others and a few do not
> encounter
>          it at all.  The duration and frequency of the outage is very
> fluid.
>          
>          During an outage, we can verify that the packet does seem to
> leave and reenter
>          the network:
>  
> Mar  5 22:28:04 pittpa-chaswv-ds3 17083: SLOT 2:6d20h:
> %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.14.174(3376) ->
> 216.41.224.3(80), 1 packet Mar  5 22:28:09 pittpa-chaswv-ds3 17084: SLOT
> 1:6d20h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
> 69.43.14.174(3376), 1 packet Mar  5 22:28:09 pittpa-chaswv-ds3 17085:
> SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp
> 69.43.14.174(3378) -> 216.41.224.3(80), 1 packet Mar  5 22:28:09
> pittpa-chaswv-ds3 17086: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP: list 111
> permitted tcp 216.41.224.3(80) -> 69.43.14.174(3378), 1 packet Mar  5
> 22:33:24 pittpa-chaswv-ds3 17089: SLOT 1:6d20h: %SEC-6-IPACCESSLOGP:
> list 111 permitted tcp 216.41.224.3(80) -> 69.43.14.174(3378), 7 packets
> Mar  5 22:33:24 pittpa-chaswv-ds3 17090: SLOT 1:6d20h:
> %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
> 69.43.14.174(3376), 17 packets Mar  5 22:33:58 pittpa-chaswv-ds3 17092:
> SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp
> 69.43.14.174(3378) -> 216.41.224.3(80), 7 packets Mar  5 22:33:58
> pittpa-chaswv-ds3 17093: SLOT 2:6d20h: %SEC-6-IPACCESSLOGP: list 113
> permitted tcp 69.43.14.174(3376) -> 216.41.224.3(80), 18 packets
>  
> Mar  5 00:58:30 pittpa-clarwv-ds3 16062: SLOT 2:5d22h:
> %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3183) ->
> 216.41.224.3(80), 1 packet Mar  5 00:58:30 pittpa-clarwv-ds3 16063: SLOT
> 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
> 69.43.23.23(3183), 1 packet Mar  5 01:03:28 pittpa-clarwv-ds3 16067:
> SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp
> 69.43.23.23(3217) -> 216.41.224.3(80), 1 packet Mar  5 01:03:28
> pittpa-clarwv-ds3 16068: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111
> permitted tcp 216.41.224.3(80) -> 69.43.23.23(3217), 1 packet Mar  5
> 01:03:34 pittpa-clarwv-ds3 16069: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP:
> list 113 permitted tcp 69.43.23.23(3228) -> 216.41.224.3(80), 1 packet
> Mar  5 01:03:34 pittpa-clarwv-ds3 16070: SLOT 1:5d22h:
> %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
> 69.43.23.23(3228), 1 packet Mar  5 01:03:39 pittpa-clarwv-ds3 16072:
> SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp
> 69.43.23.23(3239) -> 216.41.224.3(80), 1 packet Mar  5 01:03:47
> pittpa-clarwv-ds3 16073: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113
> permitted tcp 69.43.23.23(3183) -> 216.41.224.3(80), 74 packets Mar  5
> 01:04:13 pittpa-clarwv-ds3 16075: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP:
> list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3183), 72 packets
> Mar  5 01:08:46 pittpa-clarwv-ds3 16078: SLOT 2:5d22h:
> %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3218) ->
> 216.41.224.3(80), 4 packets Mar  5 01:08:46 pittpa-clarwv-ds3 16079:
> SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp
> 69.43.23.23(3217) -> 216.41.224.3(80), 3 packets Mar  5 01:08:46
> pittpa-clarwv-ds3 16080: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113
> permitted tcp 69.43.23.23(3221) -> 216.41.224.3(80), 19 packets Mar  5
> 01:08:46 pittpa-clarwv-ds3 16081: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP:
> list 113 permitted tcp 69.43.23.23(3228) -> 216.41.224.3(80), 5 packets
> Mar  5 01:08:46 pittpa-clarwv-ds3 16082: SLOT 2:5d22h:
> %SEC-6-IPACCESSLOGP: list 113 permitted tcp 69.43.23.23(3229) ->
> 216.41.224.3(80), 6 packets Mar  5 01:08:46 pittpa-clarwv-ds3 16083:
> SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113 permitted tcp
> 69.43.23.23(3236) -> 216.41.224.3(80), 9 packets Mar  5 01:08:47
> pittpa-clarwv-ds3 16084: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP: list 113
> permitted tcp 69.43.23.23(3233) -> 216.41.224.3(80), 12 packets Mar  5
> 01:08:47 pittpa-clarwv-ds3 16085: SLOT 2:5d22h: %SEC-6-IPACCESSLOGP:
> list 113 permitted tcp 69.43.23.23(3239) -> 216.41.224.3(80), 21 packets
> Mar  5 01:09:12 pittpa-clarwv-ds3 16087: SLOT 1:5d22h:
> %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
> 69.43.23.23(3239), 19 packets Mar  5 01:09:12 pittpa-clarwv-ds3 16088:
> SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp
> 216.41.224.3(80) -> 69.43.23.23(3228), 4 packets Mar  5 01:09:12
> pittpa-clarwv-ds3 16089: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111
> permitted tcp 216.41.224.3(80) -> 69.43.23.23(3217), 2 packets Mar  5
> 01:09:12 pittpa-clarwv-ds3 16091: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP:
> list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3218), 3 packets
> Mar  5 01:09:13 pittpa-clarwv-ds3 16092: SLOT 1:5d22h:
> %SEC-6-IPACCESSLOGP: list 111 permitted tcp 216.41.224.3(80) ->
> 69.43.23.23(3221), 17 packets Mar  5 01:09:13 pittpa-clarwv-ds3 16093:
> SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111 permitted tcp
> 216.41.224.3(80) -> 69.43.23.23(3229), 5 packets Mar  5 01:09:13
> pittpa-clarwv-ds3 16094: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP: list 111
> permitted tcp 216.41.224.3(80) -> 69.43.23.23(3236), 7 packets Mar  5
> 01:09:13 pittpa-clarwv-ds3 16096: SLOT 1:5d22h: %SEC-6-IPACCESSLOGP:
> list 111 permitted tcp 216.41.224.3(80) -> 69.43.23.23(3233), 9 packets
>  
>          
>          Network analysis showed significant amounts of spoofed
> multicast traffic and 
>          odd arp traffic.
>  10:17:16.416222 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 278
> 10:17:16.421886 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 334
> 10:17:16.423873 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 262
> 10:17:16.426948 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 254
> 10:17:16.432095 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 298
> 10:17:16.435921 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 274
> 10:17:16.439959 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 328
> 10:17:16.445317 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 326
> 10:17:16.449688 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 330
> 10:17:16.463537 IP 192.168.1.1.1900 > 239.255.255.250.1900: udp 322
> 
>          Steps were taken to elminiate the spoofed traffic on the 
>          routers and access servers in the form of ACLs and filter
> lists.
> 	 Neither have eliminiated the problem... but to what extent they
> might
> 	 have helped has yet to be determined.
>  
>          The problem is still occuring, for some users the duration of
> the outage 
>          seems to have shortened other users notice no difference.  It
> is not yet
>          known if the filtering on the routers and access servers or the
> conversion
>          to the 10.x.x.x network has made any difference.  We should
> have a better idea
>          in the upcoming days.
>  
> What Doesn't help: Removing windows updates.  
>                    Turning off XP firewall.
>                    Searching for malware. (SpyBot-SD, Adaware)
>                    Virus scanning (Various softwares)
>                    Specifying dns servers.
>                    Reinstalling windows.
> 

• Previous message (by thread): Weird network problem.
• Next message (by thread): thanks for the great response on wholesalebandwidth.com major abuser
• Messages sorted by: [ date ] [ thread ] [ subject ] [ author ]