real data [Re: General Internet Instability]

Mon Nov 7 17:13:25 UTC 2011

Jared,
     This is good stuff and I'm understanding how you interpret the 
data. So this confirms what we are seeing. How do we take this towards a 
root cause? Mash it with the Juniper threads and see where it goes?

-Hammer-

"I was a normal American nerd"
-Jack Herer


On 11/07/2011 11:01 AM, Jared Mauch wrote:
> Here's some real data for those interested.  It seems a quick view seems many TATA<->  Level3 and TATA<->  GBLX sets of instability.
>
> Combined with the overall update levels seen over that 30 minutes, we saw ~1.566M updates at route views.
>
> Compared with the 24h prior (2011.11.06 14:15 as reference) we see ~17-20x the updates in the same time period.  Now take your control plane and make it work 17-20x as hard, and you can see why we had some even general instability.
>
> I don't have better tools for doing a more detailed analysis at this time, but this should get you started in talking to others about what happened.  (There were no stand-out prefixes for this time-period, they all had about the same number of updates per prefix).
>
> #Updates|Filename
> --------+-----------------
>     42041 20111106.1415.out
>    727711 20111107.1400.out
>    838894 20111107.1415.out
>
> 2011.11.07 14:15 datafile - Most updates/as-path (complete as seen @ rviews)
> Count AS_PATH
> -----+----------------------------
> 4563  3549 3356
> 3172  3549 3356 11492
> 2912  8492 3356 6453 4755
> 2729  812 6453 4755
> 2695  3549 6453 14080 10620
> 2504  3549 3356 11492 11492 11492
> 2421  3549 3356 7029
> 2362  3356 209 721 27064
> 2244  3356 209 20115
> 2130  3356 209 22561
> 2044  7018 6453 14080 10620
> 2000  3356 209
> 1934  3549 3356 2907
> 1909  3356 15412 18101
> 1821  3549 6453 4755
> 1807  3356 6453 4755
> 1660  8492 3356 174
> 1634  3549 3356 18566
> 1619  3549 3356 4837 17623
> 1617  7018 6453 4755
> 1581  2497 6453 7545 7545 7545
> 1496  8492 3356 209 721 27064
> 1478  2497 6453 4755
> 1437  8492 8928 3356 11492 11492 11492
> 1427  3356 3549 3216 8402
> 1395  3356 701 19262
> 1362  8492 3356 209 20115
> 1357  3549 7843 7843 7843 10796
> 1336  3549 3356 6830 6830 6830
> 1259  2497 3356
> 1238  3356 9498
> 1229  812 6453 14080 10620
> 1229  3356 6453 14080 10620
> 1221  3356 9121 42926
> 1220  3549 3356 29314
> 1203  3549 3356 680 680 680 680 680
> 1194  2497 3356 5650 7011
> 1165  2497 3356 9498
> 1151  8001 4436 7843 10796
> 1150  3549 3356 15412 18101 18101 18101 18101 17803
> 1106  8492 3356 6762 7303
> 1105  3549 3356 55410 45528
> 1098  3549 3356 15412 18101 17803
> 1073  3549 7843 7843 7843
> 1072  3549 3356 7713 17974
> 1069  3549 3356 15290
> 1041  3356 3549
> 1032  3549 3356 11992
> 1030  3356 4323
> 1021  3549 6453 7843 11351
> 1020  8492 8928 3356 7843 11351
> 1016  2497 3356 11492
> 1014  3356 2907
> 1011  7018 3356 2907
>
>
>
> 2011.11.07 14:00 datafile - Most updates/as-path (complete as seen @ rviews)
> Count AS_PATH
> -----+----------------------------
> 3894  3356 6453 4755
> 3504  3549 3356
> 2930  812 6453 4755
> 2793  3549 6453 4755
> 2254  3549 3356 11492 11492 11492
> 2235  812 6453 14080 10620
> 1826  2497 6453 4755
> 1795  3549 3356 7029
> 1753  3356 3549
> 1689  7018 6453 14080 10620
> 1653  812 6453 4755 17488
> 1649  3549 3356 18566
> 1613  2497 3356
> 1568  3549 3356 6830 6830 6830
> 1496  7018 6453 4755
> 1419  3549 6453 14080 10620
> 1394  3356 701 19262
> 1389  3356 9121 42926
> 1382  3549 3356 21565
> 1293  3549 6453 4755 17488
> 1252  3549 3356 13407
> 1188  2497 3356 5650 7011
> 1156  7018 6453 4755 17488
> 1154  3356 6453 14080 10620
> 1127  3356 701 3216 8402
> 1106  8492 9002 3549 6762 7303
> 1104  3549 3356 4837 17623
> 1049  39756 3356 7018
> 1048  39756 3257 7018
> 1035  2497 6453 7713 17974
> 1008  3356 3320
>
>
>