The Making of a Router
rps at maine.edu
Fri Dec 27 10:14:39 UTC 2013
In talking about RAMBOOT I also realized the instructions are out of date
on the website.
The "ramboot" boot target script was updated since I created the initial
page to generate the correct fstab, and enable the root account, set a
hostname, etc. so you can actually use the OS until you create a new image.
I extracted the script from the initird to make it easier to grab:
Essentially, by adding a new "ramboot" script to
"/usr/share/initramfs-tools/scripts" along side "nfs" and "local" it
creates a new "boot=" target (since the init script looks for
As mentioned on the website, the ramboot process needs a more complete
version of busybox (for tar archive support) and the mke2fs tool added to
"/usr/lib/initramfs-tools/bin/" so they will be available to the initrd.
Once you configure networking (see the "INSTALL/setup_network" script) you
can do an apt-get update and apt-get the packages you need from Ubuntu
Example starting point:
apt-get install sudo
apt-get install nano
apt-get install ssh
apt-get install vlan
apt-get install bridge-utils
On Thu, Dec 26, 2013 at 8:27 PM, Ray Soucy <rps at maine.edu> wrote:
> The basic idea of RAMBOOT is typical in Embedded Linux development.
> Linux makes use of multi-stage boot process. One of the stages involves
> using an initial ramdisk (initrd) to provide a base root filesystem which
> can be used to locate and mount the system root, then continue the boot
> process from there.
> For an in-memory OS, instead of the boot process mounting a pre-loaded
> storage device with the expected root filesystem (e.g. your installed HDD),
> you modify it to:
> 1) Create and format a ramdisk.
> 2) Create the expected directory structure and system files for the root
> filesystem on that ramdisk.
> The root filesystem includes the /dev directory and appropriate device
> nodes, the basic Linux filesystem and your init program.
> The easy way to do that is just to have a TAR archive that you extract to
> the ramdisk on boot, better yet use compression on it (e.g. tar.gz) so that
> the archive can be read from storage (e.g. USB flash) more quickly.
> Today, the initramfs in Linux handles a lot more than simply mounting the
> storage device. It performs hardware discovery and loads appropriate
> modules. As such the Debian project has a dynamic build system for
> initramfs that is run to build the initrd when a new kernel package is
> installed, it's called "initramfs-tools".
> You can manually build your own initramfs using the examples on the
> RAMBOOT website, but the point of RAMBOOT is to make building an in-memory
> OS quick and simple.
> RAMBOOT instead adds configuration to initramfs-tools so that each time a
> new initrd is generated, it includes the code needed for RAMBOOT.
> The RAMBOOT setup adds handling of a new boot target called "ramboot" to
> the kernel arguments. This allows the same kernel to be used for a normal
> installation and remain unaffected, but when you add the argument
> "boot=ramboot" as a kernel option to the bootloader, it triggers the
> RAMBOOT process described above.
> Having a common kernel between your development environment and embedded
> environment makes it much easier to test and verify functionality.
> The other part of RAMBOOT is that it makes use of "Ubuntu Core". Ubuntu
> Core is a stripped down minimal (and they really do mean minimal) root
> filesystem for Embedded Linux development. It includes apt-get, though, so
> you can install all the packages you need from Ubuntu on the running system.
> RAMBOOT then has a development script to make a new root filesystem
> archive with the packages you've installed as a baseline. This allows for
> you to boot a RAMBOOT system, install your desired packages and change
> system configuration files as desired, then build a persistent image of
> that install that will be used for future boots.
> I also have the start of a script to remove unused kernel modules, and
> other files (internationalization for example) which add to the OS
> You could build the root filesystem on your own (and compile all the
> necessary packages) but using Ubuntu Core provides a solid base and allows
> for the rapid addition of packages from the giant Ubuntu repository.
> Lastly, I make use of SYSLINUX as a bootloader because my goal was to use
> a USB stick as the bootflash on an Atom box. Unfortunately, the Atom BIOS
> will only boot a USB device if it has a DOS boot partition, so GRUB was a
> no-go. The upside is that since the USB uses SYSLINUX and is DOS
> formatted, it's easily mounted in Windows or Mac OS X, allowing you to copy
> new images or configuration to it easily.
> For the boot device I make use of the on-board vertical USB socket on the
> system board (typical for most system boards these days) and a low-profile
> USB stick. I find the Verbatim "Store 'n' Go" 8GB USB stick ideally suited
> for this as it's less than a quarter-inch high after the USB adapter.
> RAMBOOT as a project is in the very early stages, so you should be
> comfortable with Linux before you build a system on it. And I really feel
> it's more of an example than anything at this point.
> There are several advantages though:
> The most common point of failure on a Linux system is the storage device
> (either HDD or SSD).
> The biggest bottleneck in system performance is storage IO.
> Using a ramdisk eliminates both these concerns (in fact, even an Atom
> system has surprisingly great performance when run using a ramdisk).
> The result is that you get a very reliable, high-performance system.
> The other benefit to RAMBOOT is that the root filesystem is NOT
> persistent. This means that like a Cisco device, every boot of the system
> brings you to a known working state OS-wise. There are hundreds of system
> files in a Linux system; any one of which being modified could cause
> problems. For both security and availability concerns a lot of effort is
> invested in detecting changes to system files and avoiding them. With
> RAMBOOT the problem is easily avoided.
> A minimal system can fit within a 512MB ramdisk. But with RAM being so
> cheap these days, I think even reserving up to 2GB of RAM for a ramdisk
> would be fine (e.g. for a 4-8GB system).
> Here is the hardware configuration I originally started RAMBOOT for in
> 2011 (wanted to avoid the cost of an HDD):
> $326.36 (shipping included):
> 1U rack-mount case Supermicro CSE-502L-200B
> Intel Atom D510 system board with dual Gigabit (Intel 82574L) Supermicro
> 2GB RAM
> 8GB low-profile USB flash drive (which will connect to the internal USB
> port and be low enough to fit in the case) Verbatim "Store n Stay".
> No HD; the system will boot off the 8GB flash into RAM and run the OS on a
> Using the RAMBOOT release that's currently up, I can build a custom Linux
> in-memory OS in half a day. I can easily update packages for security
> updates from the Ubuntu project and re-generate a new, updated, image in
> less time than that. So the initial goal of being able to build something
> useful quickly was satisfied, at least. My attention has now moved on to
> building a configuration management system, similar to Vyatta or VyOS and
> building a real distribution. I was going to call it "Carrier-grade Linux"
> (cglinux.org), but given the momentum VyOS has I might try to help the
> VyOS community instead of doing something new on my own.
> For what it's worth, I'm actually working with the VyOS project to try and
> incorporate some of the RAMBOOT ideas into VyOS as an install option for
> in-memory only.
> If you make use of RAMBOOT I would love to hear about it. :-)
> On Thu, Dec 26, 2013 at 4:22 PM, Nick Cameo <symack at gmail.com> wrote:
>> Inline response exist,
>> On 12/26/13, Ray Soucy <rps at maine.edu> wrote:
>> > You can build using commodity hardware and get pretty good results.
>> > I've had really good luck with Supermicro whitebox hardware, and
>> > Intel-based network cards. The "Hot Lava Systems" cards have a nice
>> > selection for a decent price if you're looking for SFP and SFP+ cards
>> > use Intel chipsets.
>> I like the supermicro as well however we have a couple of IBM x3250
>> with 2 pcie v3
>> x8 that are begging for a intel network card.
>> > There might be some benefits in going with something like FreeBSD, but I
>> > find that Linux has a lot more eyeballs on it making it much easier to
>> > develop for, troubleshoot, and support. There are a few options if you
>> > want to go the Linux route.
>> This is very important to consider. I would be speculating, or even
>> worse, expecting
>> the same type of community support from the BSD verse that I have been
>> getting from the linux community.
>> > Option 1: Roll your own OS. This takes quite a bit of effort, but if
>> > have the tallant to do it you can generally get exactly what you want.
>> If Free/OpenBSD is ruled out, I could crack open the LFS project. You only
>> have to do it once right? Or maybe just reach out to the gentoo community
>> for a stripped version, and build outwards.
>> > The biggest point of failure I've experienced with Linux-based routers
>> > whitebox hardware has been HDD failure. Other than that, the 100+ units
>> > I've had deployed over the past 3+ years have been pretty much flawless.
>> > Thankfully, they currently run an in-memory OS, so a disk failure only
>> > affects logging.
>> > If you want to build your own OS, I'll shamelessly plug a side project
>> > mine: RAMBOOT
>> > http://ramboot.org/
>> > RAMBOOT makes use of the Ubuntu Core rootfs, and a modified boot process
>> > (added into initramfs tools, so kernel updates generate the right kernel
>> > automatically). Essentially, I use a kernel ramdisk instead of an HDD
>> > the root filesystem and "/" is mounted on "/dev/ram1".
>> > The bootflash can be removed while the system is running as it's only
>> > mounted to save system configuration or update the OS.
>> > I haven't polished it up much, but there is enough there to get going
>> > pretty quickly.
>> Ummm, if it's ok with the community, can you kindly elaborate :). I am
>> not too fond of Debian since my horrible experience with Squeeze Desktop.
>> I would maybe like to try this using the combination of SSD, in memory,
>> > You'll also want to pay attention to the settings you use for the
>> > Linux is tuned as a desktop or server, not a router, so there are some
>> > basics you should take care of (like disabling ICMP redirects,
>> > the ARP table size, etc).
>> Totally strip it as much as possible. If anyone has a Gentoo stripped
>> config that they would like to share, please do :).
>> > I have some examples in: http://soucy.org/xorp/xorp-1.7-pre/TUNING
>> > or http://soucy.org/tmp/netfilter.txt (more recent, but includes
>> > examples).
>> Will definitely look into all your sites.
>> > Also a note of caution. I would stick with a longterm release of Linux.
>> > I've had good experience with 2.6.32, and 3.10. I'm eager to use some
>> > the post-3.10 features, though, so I'm anxious for the next longterm
>> > to be locked in.
>> We are comfy with 3.4 right now...
>> > One of the biggest advantages is the low cost of hardware allows you to
>> > maintain spare systems, reducing the time to service restoration in the
>> > event of failure. Dependability-wise, I feel that whitebox Linux
>> > are pretty much at Cisco levels these days, especially if running
>> > in-memory.
>> Really interested with the "in-memory", however, I would love to implement
>> it using gentoo as mentioned above.
>> Kind Regards,
> Ray Patrick Soucy
> Network Engineer
> University of Maine System
> T: 207-561-3526
> F: 207-561-3531
> MaineREN, Maine's Research and Education Network
Ray Patrick Soucy
University of Maine System
MaineREN, Maine's Research and Education Network
More information about the NANOG