Reducing Usenet Bandwidth

Joe St Sauver JOE at OREGON.UOREGON.EDU
Sat Feb 9 18:34:00 UTC 2002


>Date: Fri, 08 Feb 2002 22:09:12 -0800
>From: Stephen Stuart <stuart at tech.org>
>Subject: Re: Reducing Usenet Bandwidth
>To: David Schwartz <davids at webmaster.com>
>Cc: nanog at merit.edu

[snip]

>Some of those problems in that vein that appeared insurmountable ten years 
>ago might have solutions in current "peer-to-peer" networking technologies
>(thus the off-hand comment about Napster).

What you're really asking for is a NNTP-over-FastTrack (Kazaa/Morpheus) feed
object. This is sort of/really an odd idea, but if you think about it, all 
the required parts are pretty easy to articulate in skeleton form:

1) Each file dumped into FastTrack/Kazaa/Morpheus would get as its "filename"
the article's message-ID. A site that wanted to retrieve an article would 
issue (via FastTrack/Kazaa/Morpheus) a search request for that message-ID, 
retrieve the article from a FastTrack/Kazaa/Morpheus server that offered that
file, and then add it to the local news news server spool (and/or, optionally,
a FastTrack/Kazaa/Morpheus shared directory on the local server). 

2) How would servers know what article IDs to ask for? Well, Usenet overview 
data for each group could also be published by news servers via 
FastTrack/Kazaa/Morpheus with filenames of the format 
pathtag!hierarchy.subhierarchy.group!yy:mm:dd:hh:mm:ss

The pathtag would be required because obviously different news servers will
know about different articles, and you will want to disambiguate which 
server's overview data you want (not because there's any expectation that 
any news server will in fact provide public access to any articles, but 
simply because one might expect that N servers might all simultaneously offer 
comparable but non-identical overview data for any given group, and you might
want to choose the view from server foo, or bar, or baz over other 
alternatives). 

The hierarchy.subhierarchy.group component of the overview file name simply 
allows folks to identify the overview data file they'd want.

The date/time stamp (GMT) provides a means of selecting the most recently 
available overview data from a list of matching responses.

Overview data would obviously need to be cryptographically signed with a 
key tied to the pathtag to avoid problems with people providing forged 
overview data (wouldn't want people to be fed lists of bogus/inappropriate 
lists of message ID's as a DOS/spam attack, right?).

Overview files could be generated at some server-determined periodicity, and
to avoid inspiring any sort of synchronization effect, I'll omit mentioning
a value here except to say that I've got something in the double-digits-of-
minutes in mind here. (Besides, if you're going to publish overview data for
some tens of thousands of groups, it will take a while to walk that tree). 

3) Client servers interested in retrieving articles via 
FastTrack/Kazaa/Morpheus could routinely retrieve overview records for 
groups of interest, retrieving all/some of the articles as a "prefetch" 
for popular groups routinely read by their users, or the server could wait
until articles have been requested, then retrieve an overview file, then 
retrieve articles. 

Alternatively, individual users with "News over FastTrack/Kazaa/Morpheus" 
clients could be retrieving articles directly from Kazaa, removing any need 
for an intervening news server whatsoever (although hopefully a local 
"regular" news server would always be faster than a distributed 
FastTrack/Morpheus/Kazaa-based news spool). 

Regards,

Joe

P.S. Needless to say, this would really change the FastTrack/Kazaa/Morpheus
content base, and given the volumes Usenet is currently transferring, would
be sort of an interesting experiment if we're talking about a full feed...



More information about the NANOG mailing list