The Making of a Router

Fri Dec 27 01:27:03 UTC 2013

The basic idea of RAMBOOT is typical in Embedded Linux development.

Linux makes use of multi-stage boot process.  One of the stages involves
using an initial ramdisk (initrd) to provide a base root filesystem which
can be used to locate and mount the system root, then continue the boot
process from there.

For an in-memory OS, instead of the boot process mounting a pre-loaded
storage device with the expected root filesystem (e.g. your installed HDD),
you modify it to:

1) Create and format a ramdisk.
2) Create the expected directory structure and system files for the root
filesystem on that ramdisk.

The root filesystem includes the /dev directory and appropriate device
nodes, the basic Linux filesystem and your init program.

The easy way to do that is just to have a TAR archive that you extract to
the ramdisk on boot, better yet use compression on it (e.g. tar.gz) so that
the archive can be read from storage (e.g. USB flash) more quickly.

Today, the initramfs in Linux handles a lot more than simply mounting the
storage device.  It performs hardware discovery and loads appropriate
modules.  As such the Debian project has a dynamic build system for
initramfs that is run to build the initrd when a new kernel package is
installed, it's called "initramfs-tools".

You can manually build your own initramfs using the examples on the RAMBOOT
website, but the point of RAMBOOT is to make building an in-memory OS quick
and simple.

RAMBOOT instead adds configuration to initramfs-tools so that each time a
new initrd is generated, it includes the code needed for RAMBOOT.

The RAMBOOT setup adds handling of a new boot target called "ramboot" to
the kernel arguments.  This allows the same kernel to be used for a normal
installation and remain unaffected, but when you add the argument
"boot=ramboot" as a kernel option to the bootloader, it triggers the
RAMBOOT process described above.

Having a common kernel between your development environment and embedded
environment makes it much easier to test and verify functionality.

The other part of RAMBOOT is that it makes use of "Ubuntu Core".  Ubuntu
Core is a stripped down minimal (and they really do mean minimal) root
filesystem for Embedded Linux development.  It includes apt-get, though, so
you can install all the packages you need from Ubuntu on the running system.

RAMBOOT then has a development script to make a new root filesystem archive
with the packages you've installed as a baseline.  This allows for you to
boot a RAMBOOT system, install your desired packages and change system
configuration files as desired, then build a persistent image of that
install that will be used for future boots.

I also have the start of a script to remove unused kernel modules, and
other files (internationalization for example) which add to the OS
footprint.

You could build the root filesystem on your own (and compile all the
necessary packages) but using Ubuntu Core provides a solid base and allows
for the rapid addition of packages from the giant Ubuntu repository.

Lastly, I make use of SYSLINUX as a bootloader because my goal was to use a
USB stick as the bootflash on an Atom box.  Unfortunately, the Atom BIOS
will only boot a USB device if it has a DOS boot partition, so GRUB was a
no-go.  The upside is that since the USB uses SYSLINUX and is DOS
formatted, it's easily mounted in Windows or Mac OS X, allowing you to copy
new images or configuration to it easily.

For the boot device I make use of the on-board vertical USB socket on the
system board (typical for most system boards these days) and a low-profile
USB stick.  I find the Verbatim "Store 'n' Go" 8GB USB stick ideally suited
for this as it's less than a quarter-inch high after the USB adapter.

RAMBOOT as a project is in the very early stages, so you should be
comfortable with Linux before you build a system on it.  And I really feel
it's more of an example than anything at this point.

There are several advantages though:

The most common point of failure on a Linux system is the storage device
(either HDD or SSD).
The biggest bottleneck in system performance is storage IO.
Using a ramdisk eliminates both these concerns (in fact, even an Atom
system has surprisingly great performance when run using a ramdisk).

The result is that you get a very reliable, high-performance system.

The other benefit to RAMBOOT is that the root filesystem is NOT persistent.
 This means that like a Cisco device, every boot of the system brings you
to a known working state OS-wise.  There are hundreds of system files in a
Linux system; any one of which being modified could cause problems.  For
both security and availability concerns a lot of effort is invested in
detecting changes to system files and avoiding them.  With RAMBOOT the
problem is easily avoided.

A minimal system can fit within a 512MB ramdisk.  But with RAM being so
cheap these days, I think even reserving up to 2GB of RAM for a ramdisk
would be fine (e.g. for a 4-8GB system).

Here is the hardware configuration I originally started RAMBOOT for in 2011
(wanted to avoid the cost of an HDD):

$326.36 (shipping included):
1U rack-mount case Supermicro CSE-502L-200B
Intel Atom D510 system board with dual Gigabit (Intel 82574L) Supermicro
MBD-X7SPA-H-O
2GB RAM
8GB low-profile USB flash drive (which will connect to the internal USB
port and be low enough to fit in the case) Verbatim "Store n Stay".

No HD; the system will boot off the 8GB flash into RAM and run the OS on a
ramdisk.

Using the RAMBOOT release that's currently up, I can build a custom Linux
in-memory OS in half a day.  I can easily update packages for security
updates from the Ubuntu project and re-generate a new, updated, image in
less time than that.  So the initial goal of being able to build something
useful quickly was satisfied, at least.  My attention has now moved on to
building a configuration management system, similar to Vyatta or VyOS and
building a real distribution.  I was going to call it "Carrier-grade Linux"
(cglinux.org), but given the momentum VyOS has I might try to help the VyOS
community instead of doing something new on my own.

For what it's worth, I'm actually working with the VyOS project to try and
incorporate some of the RAMBOOT ideas into VyOS as an install option for
in-memory only.

If you make use of RAMBOOT I would love to hear about it. :-)

On Thu, Dec 26, 2013 at 4:22 PM, Nick Cameo <symack at gmail.com> wrote:

> Inline response exist,
>
> On 12/26/13, Ray Soucy <rps at maine.edu> wrote:
> > You can build using commodity hardware and get pretty good results.
> >
> > I've had really good luck with Supermicro whitebox hardware, and
> > Intel-based network cards.  The "Hot Lava Systems" cards have a nice
> > selection for a decent price if you're looking for SFP and SFP+ cards
> that
> > use Intel chipsets.
>
> I like the supermicro as well however we have a couple of IBM x3250
> with 2 pcie v3
> x8 that are begging for a intel network card.
>
> > There might be some benefits in going with something like FreeBSD, but I
> > find that Linux has a lot more eyeballs on it making it much easier to
> > develop for, troubleshoot, and support.  There are a few options if you
> > want to go the Linux route.
>
> This is very important to consider. I would be speculating, or even
> worse, expecting
> the same type of community support from the BSD verse that I have been
> getting from the linux community.
>
> >
> > Option 1: Roll your own OS.  This takes quite a bit of effort, but if you
> > have the tallant to do it you can generally get exactly what you want.
>
> If Free/OpenBSD is ruled out, I could crack open the LFS project. You only
> have to do it once right? Or maybe just reach out to the gentoo community
> for a stripped version, and build outwards.
>
> > The biggest point of failure I've experienced with Linux-based routers on
> > whitebox hardware has been HDD failure.  Other than that, the 100+ units
> > I've had deployed over the past 3+ years have been pretty much flawless.
> >
>
> SSD
>
> > Thankfully, they currently run an in-memory OS, so a disk failure only
> > affects logging.
> > If you want to build your own OS, I'll shamelessly plug a side project of
> > mine: RAMBOOT
> >
> > http://ramboot.org/
> >
> > RAMBOOT makes use of the Ubuntu Core rootfs, and a modified boot process
> > (added into initramfs tools, so kernel updates generate the right kernel
> > automatically).  Essentially, I use a kernel ramdisk instead of an HDD
> for
> > the root filesystem and "/" is mounted on "/dev/ram1".
> >
> > The bootflash can be removed while the system is running as it's only
> > mounted to save system configuration or update the OS.
> >
> > I haven't polished it up much, but there is enough there to get going
> > pretty quickly.
>
> Ummm, if it's ok with the community, can you kindly elaborate :). I am
> not too fond of Debian since my horrible experience with Squeeze Desktop.
> I would maybe like to try this using the combination of SSD, in memory, and
> Gentoo?
>
> >
> > You'll also want to pay attention to the settings you use for the kernel.
> >  Linux is tuned as a desktop or server, not a router, so there are some
> > basics you should take care of (like disabling ICMP redirects, increasing
> > the ARP table size, etc).
>
> Totally strip it as much as possible. If anyone has a Gentoo stripped
> kernel
> config that they would like to share, please do :).
>
> >
> > I have some examples in: http://soucy.org/xorp/xorp-1.7-pre/TUNING
> > or http://soucy.org/tmp/netfilter.txt (more recent, but includes
> firewall
> > examples).
>
> Will definitely look into all your sites.
>
> >
> > Also a note of caution.  I would stick with a longterm release of Linux.
> >  I've had good experience with 2.6.32, and 3.10.  I'm eager to use some
> of
> > the post-3.10 features, though, so I'm anxious for the next longterm
> branch
> > to be locked in.
> >
>
> We are comfy with 3.4 right now...
>
>
> > One of the biggest advantages is the low cost of hardware allows you to
> > maintain spare systems, reducing the time to service restoration in the
> > event of failure.  Dependability-wise, I feel that whitebox Linux systems
> > are pretty much at Cisco levels these days, especially if running
> > in-memory.
>
> Really interested with the "in-memory", however, I would love to implement
> it using gentoo as mentioned above.
>
>
> Kind Regards,
>
> N.
>

-- 
Ray Patrick Soucy
Network Engineer
University of Maine System

T: 207-561-3526
F: 207-561-3531

MaineREN, Maine's Research and Education Network
www.maineren.net