2006.06.07 NANOG-NOTES Lightning talk notes

Fri Jun 9 22:49:50 UTC 2006

(I think these were the toughest to take notes on, since they went
by so fast; took the most cleaning up afterwards.  But they were also
the best talks of the 3 days.  I wish we could have flipped, and taken
more time on Tuesday for them so we really could have dug in and
asked the questions we were itching to ask.  ^_^;  --Matt)

2006.06.07 Lightning talks

Marty Hannigan, Renesys:
[slides are at:
http://www.nanog.org/mtg-0606/pdf/lightning-talks/1-hannigan.pdf

Critical infrastructure, root server location
analysis
Where to stick your servers.  :)
he took some public info out there on root-servers.org
talked to some people, extrapolated from larger set of
data.
operator demographics.
in  US:
3 corp a, c, j
2 edu b and d
1 mil g
2 research e/h
3 nonprofit f, i, l
 autonomica is responsible for l, but hosts "some"
 instances on a CDN; CDN is a US formed entity
in EU:
 1 non profit k
asia/japan:
 1 nonprofit m

92% of system operated in US, 8% non-us;
5% margin of error +-.
US entity type
non-us 8%
us corp 39%
us mil 23%
us edu 15%
us nonprofit 15%

where?
in 54 countries
 all religions
 all methods of governance

politically:
79% are democratic governments
21% in other forms of government

global diversification for security and performance
 instances spread across continents
 different networks
 different proceedures
 different software
 different hardwware
 different weaknesses
  weaknesses become strength, since they are diverse;
  no one weakness knocks out all servers.
  little less open to insider malfeasance

Global distribution
NA 38%
EU 35%
Asia 12%
AUs 8%
east EU 3%
LA 2%
Africa 2%
ANT 0%

getting reasonable coverage in the world

situating a root server
relationship 101
 who you know
  ICANN, operator, IX, and RIR relationships
  regulators
 how you spin it
  national pride
  performance and security
  betterment of user experience

Threats
 no different from anyone else
  direct attacks
  proxy attacks
  botnets
   easy money
   miscreants masking other activities

Not sure what motivations to attack root servers;
can't extort money from nonprofits

let's attack a root server
target $-root
  location; eu hosting facility
  multi-post cabinet config with cabling and power
   under floor
  unlocked cabinet, single factor facility entry
 physical attack
   open cabinet door
   access to power
hijack attempt
  advertise a route
  return bad answers
network attack
 spoof source
 random host queries
 packet floods

summary:
 root system is less likely to be subject to insider
  attack or weakness
 but can be attacked by layer 3
there is likely good resarch data coming across those
  interfaces
trend towards a collapsed root system, where root and
 TLD share same hardware or networks should be more
 closely examined.
slides will be up soon, talk to him in the hallway

NEXT, Anton Kapela
Network RTTs
[slides are at:
http://www.nanog.org/mtg-0606/pdf/lightning-talks/2-kapela.pdf

I'm pinging 10: high rate active probes
we're pinging stuff really quickly
adjusted host kern.hz to 1000 select() gets pretty
  accurate +-1ms emmission accuracy
stuff is responding
Interesting 0.001% of data relates to end-to-end queuing

what has been sampled?
some cisco 7513s
IOS 12.3 mainline
linux 2.4.20
freebsd 4.8
NT4 sp6
various end-to-end paths on u-wisc network

raw data isn't terrible interesting.
in adaptive link layer protocols, see rate shifting
 manifested in RTT
wireless, HPNA/HCNA, powerline ethernet
10,30,60,90 second peaks

fourier transforms, wavelet transforms, frequency domain
1000 seconds at 10ms intervals
break into composite, aggregate graph at top,
0-50hz span on x axis, y axis is contribution
summary of entire graph.
bottom right graph is rough 200 samples of a
range from 0-5hz, 100pps, deduce delay at half
that sampling rate.

delay is not a simple boring thing; has
scheduler delays, path dynamics not visible
before to see queue depths.

shark fins showed up; congestion events do
occur, are quite measurable.
when links are hot, queues are obvious, esp. on
highly multiplexed links.

bottom left, cubic resonance, several tens of
thousands of multiplexed flows hitting odd
resonance.

pinging windows machine, composite spectral
fingerprint; 10,20,25,30 spikes
Linux fewer spikes
freebsd low and flat
IOS is 10, 20, 30 and grass of 1hz spacing
 below 10hz.

win32 delay spectrum also has 1hz fuzz below
10hz.

Sampled RTT and performed signal analysis of it;
now what?
is network time continuous? is round trip time
 discreet or continous?
no changes in revealed as you go down lower
is delay a "signal' anyway
what's with the 0 hz DC component in the FT output?

could this be used for fingerprinting?
yes, could be like next nmap.
packet-level fingerprinting is trivial to fake; but
 IP stack scheduler behaviour doesn't change so
 easily.

NEXT:
Mikael Abrahamsson
Affect on traffic from the TPB bust
with Kurtis Lindqvist
[slides are at:
http://www.nanog.org/mtg-0606/pdf/lightning-talks/3-abrahamsson.pdf

Bittorrent background
p2p protocol for filesharing.
text string, upload to tracker, get IPs of other clients
 that have done the same thing, clients connect to each
  other, develop a swarm.
 clients communicate even when tracker vanishes.
  just can't get new clients joining

Thepiratebay.org
 run by a handful of individuals aged 22 to 28
 used ~100mb at peak
 peaked at 2M concurrent users
 stats code in tracker indicated that total p2p
  traffic was close to 100gig/sec
 thus far, largest bittorrent site/tracker in world

photo slide showing the physical gear
10 high-end small servers in half a rack in stockholm,
sweden
web frontends, db servers, trackers

on the stats
 not an exact science
 at least a german ISP had an outage at the same time
 bust was around 12.00 CET may 31st (euro time)
 data collected from Euro-IX members

some saw no difference.
Netnod aggregated, biggest drop, about 10+Gb drop
 very quickly
in Netnod stockholm *very* visible.
stats server was slashdotted, lost an hour of stats.
LINX London, saw about 5Gb drop out of 80Gb
AMSIX dropped about 5Gb out of 160Gb

DECIX frankfurt, germany, drop before noon,
FCIX, helsinki, Finland
drop fairly visibile
NIX, in norway, drop also visible.
doesn't show private exhanges/private peerings
Brussels (BNIX) also saw drop.

netflow export from big US ISP,
large chunk of bittorrent traffic packets faded off.

Thepiratesbay.org was back online 72 hours later in
Amsterdam, Netherlands
and traffic started coming back
June 6th is a holiday, watch the stats this coming
week.  :)

Aftermath
Police took ALL hosted equipment at the same site
by the same hosting company (small one, only a few
racks), caused quite a few community web sites to
go down plus commercial customers
Has spawned a lot of discussion in Sweden regarding
all issues involved.  Front page material every
day, even video surveillance of the raid from
surveillance cameras has been posted on youtube.com
Accusations of police/politicians being influenced
by White House and MPAA and others

Q: Bill Norton: what about other tracker sites, why
didn't traffic just shift to them?
A: some did, but torrent files have the tracker hard
coded in them, so they can't just flip over to other
tracker sites on their own.

Q: Roland Dobbins, back up in several countries now
including Russia, is traffic back?
A: Keep watching the graphs.
And if you want to see the bust, search for
"pirate bay" and "police", there's one link on youtube.

NEXT:
Alex Pilosov/Pilosoft
Adam Rothschild/Voxel
Nathan Patrick/Sonic.net
[slides are at:
http://www.nanog.org/mtg-0606/pdf/lightning-talks/4-pilosov.pdf

Passive Metro WDM
how it works
single mode fiber: mutiple wavelengths
also called "colours" or "lambdas"
coexisting separately
pluggable optics as enabler
low cost for passive optical equipment, particularly
 grey market

Dark fiber IRUs are very cheap.
low opex/capex

how does it work?
O band Original     1260 - 1360
E band Extended     1360 - 1460
S band Short        1460 - 1530
C band Conventional 1530 - 1565
L band Long         1565 - 1625

implementation options
active WDM cisco 15xxx, cienna, movaz, others
passive WDM using optical filters
  self-assembled patch panels
  complete systems (CUBO)

pictures of components

GWDM/WWDM
wideband multiplexing (1350/1550)
2GE fdx per pair, 1 GE fdx per strand
single strand networking the receiver is *always*
 wideband
low cost for transcievers (LX/ZX, <$500)
10GE possible (ER/LR)

Active xWDM
 beyond this scope
 everyone knows how to do it, it just costs more.

Passive CWDM
wavelength, wide channels, 8 channels
1470-1610 conventional
1270-1470 low range
cost is cheap ~$1000 per strand per end for
 CUBOs,
$300-$1000 per GBIC depending on quality
(CUBO, Taiwanese hw manufacturers)
no Xenpaks, GBICs only
20nm channel spacing
low availability on 'low range' GBICs/SFPs

Passive DWDM
each channel is narrow
0.8nm == 24 dense channel per single coarse channel
160 channels easily
25Ghz spacing
research at 12.5Ghz
Xenpaks available $9k+
few GBICs at $1500+

Filters:
build/add as you grow by mixing and matching
available in various ranges (center wavelength,
bandpass width)
Going from GWDM to GWDM/CWDM to GWDM/CWDM/DWDM

Testing and management
optical power meter
communication is key
 OOB access: HOOTS, cell phone
 you need to talk site-to-site to coordinate
  make sure cell phones don't depend on fiber
optical power monitoring/APD receivers in GBICs
 (show interface blah trans)
spectrum analyzer

Caveats
few complete commercial systems available
systems require clue and duct tape to put together
need to tune with attenuators if signal is too
 strong, attenuators differ with wavelength
flaky GBIC/SFP vendors
small-time passive optical vendors
expensive equipment for testing (spectrum analyzer,
light sources, etc)
lack of operational expertise (get hit by a bus)

Exotic options
Circulators (same wave both ways)
Interleavers (half the light, double the waves)
CWDM light into DWDM channel (similar to above)
10GE LX4/LR multiplexing

Simple Economics
2GE GWDM ~$1k
8GE CWDM ~$5k-10k
2*10GE ~$5k-10k
N*10GE DWDM ~N*$10k
prices include passive and active components,
 per end, fdx over one pair
Prices an order of magnitude lower than commercial
 systems from Cienna, Cisco.

List of vendors
Cloudy YAYA, Orient DONG,
[lots of names on slide, go read it yourself]

Questions?  mail them!
alex at pilosoft.com
asr at voxel.net
np at sonic.net

Q: Martin, what do you about timing?
A: No need for timing, each channel is separate,
  no timing needed to run this.

Q: mike hughes, linx; one thing to look at if you're
looking at GWDM/WWDM, or going bidir on one strand,
watch out for back reflections--running several channels
bidir would see itself reflected back, would declare
linkup
A: don't run two waves bidir on it--just don't do it,
it's not worth it, it's too ghetto.

NEXT:
Mohit Lad
Alerting prefix owners of hijacks in near-realtime
UCLA, joint work with a bunch of other names
[slides are at:
http://www.nanog.org/mtg-0606/pdf/lightning-talks/5-lad.pdf

PHAS project?
Three properties of a security solution
 ability to see "bad" information
 ability to distinguish between "good" and "bad" info
 incentive to fix the problem

The PHAS (prefix hijack alert system) approach
use updates from existing BGP monitors (route views and
 RIPE RIS)
 if false origination, send notification.
push complexity of detection to user
 look at email registration to decide who is allowed to
  announce prefixes.
 don't filter out false vs real changes.

PHAS origin monitor
131.179.0.0/16, UCLA block

recommend multiple email addresses, including some that
are *not* on your blocks!

Message Delivery
apply local rules before generating alarms
you shouldn't recieve duplicates of notifications
due to topological mesh-ness, it's difficult for a
hijacker to get all notifications for a block.

Evaluation: messages per AS
Dec 2005
map prefixes to origin AS using routing table
most AS receive less than 100 messages per month
  most less than 10
 local filters can limit legitimate origin changes.

readily deployable
  routeviews and RIPE RIS already collect data
alarm generation not dependent on
 cooperation from other networks
 monitoring or knowing correct origins
alarm authentication: single source
low overhead.

summary
comprehensive study using archived data
developing near-realtime system
interested in receiving notifications

send email to:
  mohit at cs.ucla.edu
  massey at cs.colostate.edu
ongoing efforts
 covered prefix hijack
 false last hop
reference:
 PHAS: usenix security 2006.

Q: Danny McPherson--that's associated with origin AS,
and origin AS could be spoofed, does it look at
combination of prefix, origin, and next hop up?
A: they are doing it on origin AS and next hop,
they'll do some more thinking about that case.

NEXT:
Rick Wesson, Support Intelligence [hehe]
Understanding abuse, aggregate it, push it back to
operators, let them know what they're doing to other
people.
[no slides, he does a live presentation of his tool]

How do I believe you?
realtime data visualization, Feb 8th, 2006
visualization.
130 different data sources, 90% passive;
10,000 domain aggregated spam trap, very
evil SMTP that filters and bans IP for some time.
1.2million events per day aggregated, about 700,000
unique IPs for the global internet.
BGP peers, aggregate based on announcements made.
Put into tool so network operators can visualize
their prefixes, drill in, and see abuse each
prefix generates.
hover over point, it shows the operator, IP address,
and what the problem was (spam, insecure web server, etc)

This shows problem areas that need to be addressed!
disseminate this information, help ISPs clean up their
networks.

Can also pass along information of abuse that has
happened to you.
If you have an AS, he can tell you what your AS has
been used for, abused for, owned, etc.

email him for more info...except he didn't list
his email info. ^_^;

Break!  short!