An appeal for more bandwidth to the Internet Archive

Wed May 13 19:00:10 UTC 2020

Thank you Jared & Tim!  Fortunately the recent upgrade has gotten us 
some overhead in the short term -- we generally run very close to the 
edge, capacity-wise, in order to get the most out of what we have, so a 
sudden unexpected burst in demand pinned us for a bit there.  I expect 
more upgrades to be necessary in the near future, particularly on the 
egress bandwidth and long-haul datacenter interconnect fronts, and would 
love to talk off-list about help with peering/pipes in the Bay Area and 
100G gear especially.

To answer a couple sibling thread questions, traffic is pretty 
legitimate -- lots of users accessing lots of content.  We're fine with 
folks downloading in bulk and most who do use our torrent functionality 
to do so.  We have a broad view of regionality of visitors, but we very 
intentionally don't keep specific usage records (after all, we are a 
library, and we don't track our patrons -- this also precludes the use 
of most commercial CDNs).  See e.g. 
https://archive.org/services/docs/api/views.html#footnote-what-is-the-deal-with-privacy-protecting-hashes-instead-of-ip-numbers-in-our-logs 
for some info.

As far as colocation costs, we run our own datacenters in the Bay Area, 
where most of our on-site staff lives and where fully ambient cooling is 
possible year-round.  Our presence in 200 Paul is a terminus for our 
dark fiber and a place to make XCs -- actually only a single rack.

Thanks again for the all the appreciation and help -

-- 
Jonah Edwards
jonah at archive.org
4157638676

On 2020-05-12 09:33, Tim Požár wrote:
> Jared...
> 
> Thanks for sharing this.  I was the first Director of Operations from
> '96 to '98, at was was then Internet Archive/Alex.  I was the network
> architect back then got them their ASN and original address space.
> Folks may also know, I help start SFMIX with Matt Peterson.
> 
> A bit more detail in this...  Some of this I got from Jonah Edwards
> who is the current Network Architect at IA.  Yes, the bottle neck was
> the line cards.  They have upgraded and that has certainly helped the
> bandwidth of late.
> 
> Peering would be a big help for IA. At this point they have two 10Gb
> LAG interfaces that show up on SFMIX that was turned up last February.
> Looking at the last couple of weeks the 95th percentile on this 20Gb
> LAG is 3 Gb.  As they just turned up on SFMIX, they are just starting
> to get peers turned up there. Eyeball networks that show up on SFMIX
> are highly encouraged to start peering with them.  Alas, they are v4
> only at this point.
> 
> Additionally, if folks do have some fat pipes that can donate
> bandwidth at 200 Paul, I am sure Jonah won't turn it down.
> 
> Tim
> 
> On 5/12/20 4:45 AM, Jared Brown wrote:
>> Hello all!
>> 
>> Last week the Internet Archive upgraded their bandwidth 30% from 47 
>> Gbps to 62 Gbps. It was all gobbled up immediately. There's a lovely 
>> solid green graph showing how usage grows vertically as each interface 
>> comes online until it too is 100% saturated. Looking at the graph 
>> legend you can see that their usage for the past 24 hours averages 
>> 49.76G on their 50G of transport.
>> 
>> To see the pretty pictures follow the below link:
>> https://blog.archive.org/2020/05/11/thank-you-for-helping-us-increase-our-bandwidth/
>> 
>> Relevant parts from the blog post:
>> "A year ago, usage was 30Gbits/sec. At the beginning of this year, we 
>> were at 40Gbits/sec, and we were handling it. ...
>> 
>> Then Covid-19 hit and demand rocketed to 50Gbits/sec and overran our 
>> network infrastructure’s ability to handle it.  So much so, our 
>> network statistics probes had difficulty collecting data (hence the 
>> white spots in the graphs).
>> 
>> We bought a second router with new line cards, and got it installed 
>> and running (and none of this is easy during a pandemic), and 
>> increased our capacity from 47Gbits/sec peak to 62Gbits/sec peak.   
>> And we are handling it better, but it is still consumed."
>> 
>> It is obvious that the Internet Archive needs more bandwidth to power 
>> the Wayback machine and to fulfill its mission of being the Internet 
>> library and the historic archive of our times.
>> 
>> The Internet Archive is present at Digital Realty SFO (200 Paul) and a 
>> member of the San Francisco Metropolitan Internet Exchange (SFMIX).
>> I appeal to all list members present or capable of getting to these 
>> facilities to peer with and/or donate bandwidth to the Internet 
>> Archive.
>> I appeal to all vendors and others with equipment that they can donate 
>> to the Internet Archive to contact them so that they can scale their 
>> services and sustain their growth.
>> 
>> The Internet Archive is currently running 10G equipment. If you can 
>> help them gain 100G connectivity, 100G routing, 100G switching and/or 
>> 100G DWDM capabilities, please reach out to them. They have the 
>> infrastructure and dark fiber to transition to 100G, but lack the 
>> equipment. You can find the Internet Archive's contact information 
>> below or you can contact Jonah at the Archive Org directly either by 
>> email or via the contact information available on his Twitter profile 
>> @jonahedwards.
>> 
>> You can also donate at https://archive.org/donate/
>> The Internet Archive is a 501(c)(3) non-profit. Donations are  
>> tax-deductible.
>> 
>> 
>> Contact information:
>> https://archive.org/about/contact.php
>> 
>> Volunteering:
>> https://archive.org/about/volunteerpositions.php
>> 
>> 
>> Disclaimer: I am not affiliated with the Internet Archive. Nobody 
>> asked me to write this post. If something angers you about this post, 
>> be angry at me. I merely think that the Internet Archive is a good 
>> thing and deserves our support.
>> 
>> Jared
>> 