Facebook Issues/Outage in Southeast?

Thu Sep 23 20:45:57 UTC 2010

Here in Alaska, we have fiber going from Anchorage down to Seattle - where we peer with Facebook via SIX. We also host localized content for akamai, who in turn caches data for facebook. Our requests to facebook.com (dns wise) are handled through the akamai cluster on our internal 10 gig link - which gives the error:
Service Unavailable - DNS failure
The server is temporarily unable to service your request. Please try again later.

Reference #11.1496a5d1.1285274423.4657e30a

-----Original Message-----
From: Jay Nugent [mailto:jjn at nuge.com] 
Sent: Thursday, September 23, 2010 11:59 AM
To: Justin Horstman
Cc: nanog at nanog.org
Subject: RE: Facebook Issues/Outage in Southeast?

Greetings,

On Thu, 23 Sep 2010, Justin Horstman wrote:

> Via http://downforeveryoneorjustme.com/facebook.com
> 
> It's not just you! http://facebook.com looks down from here.
> 
> Also down from LA, qwest has been having issues today as well, not sure if its related.

    About 45 minutes ago traceroutes died...

[jjn at K2 ~]$ traceroute www.facebook.com
traceroute to www.facebook.com (66.220.149.18), 30 hops max, 60 byte 
packets
 1  DR-LVL3.nuge.com (216.144.208.1)  1.083 ms  1.695 ms  1.934 ms
 2  * * *
 3  166.90.244.19 (166.90.244.19)  31.257 ms  34.148 ms  37.801 ms
 4  ge-5-0-155.hsa1.Detroit1.Level3.net (63.211.20.93)  41.258 ms  43.576
 5  so-8-1.car1.Detroit1.Level3.net (4.69.140.118)  51.752 ms  54.577 ms
 6  ae-8-8.ebr2.Chicago1.Level3.net (4.69.133.242)  76.007 ms  75.161 ms
 7  ae-5-5.ebr2.Chicago2.Level3.net (4.69.140.194)  76.469 ms  48.544 ms
 8  ae-6-6.ebr2.Washington12.Level3.net (4.69.148.145)  64.813 ms  63.952
 9  ae-5-5.ebr2.Washington1.Level3.net (4.69.143.221)  61.980 ms  63.616
10  ae-92-92.csw4.Washington1.Level3.net (4.69.134.158)  74.941 ms  73.267 
ms ae-72-72.csw2.Washington1.Level3.net (4.69.134.150)  80.951 ms
11  ae-2-79.edge3.Washington4.Level3.net (4.68.17.85)  74.694 ms 
    ae-1-69.edge3.Washington4.Level3.net (4.68.17.21)  108.301 ms 
    ae-2-79.edge3.Washington4.Level3.net (4.68.17.85)  80.238 ms
12  FACEBOOK-IN.edge3.Washington4.Level3.net (4.53.116.6)  110.998 ms  
13  ae2.bb02.iad2.tfbnw.net (74.119.77.148)  89.248 ms  86.623 ms  85.374
14  ae11.bb02.sjc1.tfbnw.net (74.119.77.199)  114.421 ms  117.183 ms 
15  ae1.dr01.snc5.tfbnw.net (74.119.77.182)  96.802 ms 
    ae2.dr01.snc5.tfbnw.net (74.119.77.184)  99.674 ms 
    ae1.dr02.snc5.tfbnw.net (74.119.77.190)  98.270 ms
16  po509.csw02b.snc5.tfbnw.net (74.119.78.28)  94.258 ms 
    po509.csw02a.snc5.tfbnw.net (74.119.78.24)  94.511 ms 
    po510.csw02b.snc5.tfbnw.net (74.119.78.26)  94.792 ms
17  * * *
18  * * *
19  * * *

   The 10 minutes ago the Aikami frontend answered with this:

-----------------------------------------------
Service Unavailable - Zero size object
The server is temporarily unable to service your request. Please try again 
later.

Reference #15.16ce33b8.1285271433.6b82b 
-----------------------------------------------

   Looks like the FaceBook backend is still down.  We wait (and be more 
productive in our *real* job)  :)

      --- Jay Nugent
          Nugent Telecommunications
          Ypsilanti, Michigan

Train how you will Operate, and you will Operate how you were Trained.
+------------------------------------------------------------------------+
| Jay Nugent   jjn at nuge.com    (734)484-5105    (734)649-0850/Cell       |
|   Nugent Telecommunications  [www.nuge.com]                            |
|   Internet Consulting/Linux SysAdmin/Engineering & Design/ISP Reseller |
| ISP Monitoring [www.ispmonitor.org] ISP & Modem Performance Monitoring |
| Web-Pegasus    [www.webpegasus.com] Web Hosting/DNS Hosting/Shell Accts|
+------------------------------------------------------------------------+
  3:01pm  up 15 days, 23:02,  4 users,  load average: 0.03, 0.03, 0.03