Scalability issues in the Internet routing system

Thu Oct 27 09:55:56 UTC 2005

On Thu, 27 Oct 2005 Michael.Dillon at btradianz.com wrote:

>> Another thing, it would be interesting to hear of any work on
>> breaking the "router code" into multiple threads.  Being able to
>> truly take advantage of multiple processors when receiving 2M updates
>> would be the cats pajamas.  Has anyone seen this?  I suppose MBGP
>> could be rather straightforward, as opposed to one big table, in a
>> multi-processor implementation.
>
> Why bother multithreading when you can just use multiple
> CPUs? :-)
>
> Nowadays, a CPU is not a chip, it is a core. "Core" is
> the name for a section of a chip which functions as
> a CPU. Cores are actually software written in a language
> such as VHDL (VHSIC Hardware Description Language).
> VHSIC stands for Very High Speed Integrated Circuit.
> The core is "compiled" into hardware on either an
> FPGA (Field Programmable Gate Array) or an ASIC
> (Application Specific Integrated Circuit). An FPGA
> can be reconfigured by software at any time, for instance
> you could reprogram an FPGA to do route lookups for
> a specific set of prefixes and change the hardware
> whenever the prefix list changes.
>
> Most ASICs nowadays are actually hybrid chips because
> they contain an FPGA section. Now, back to cores. Since
> the CPU core is simply software, it is possible to
> install multiple copies of the core on an FPGA or an
> ASIC if there is enough space. The cores for RISC machines
> like ARM are much smaller than the core for a Pentium
> and therefore a simple RISC CPU core can be replicated
> more times.
>
> Now, with that information in hand, you will be able
> to understand just what Cisco and IBM have done in creating
> the CRS-1 chip with a minimum of 188 CPUs on the chip.
> http://www.eet.com/showArticle.jhtml?articleID=26806315
>
> Just as the line between routers and switches has become
> blurred, so to has the line between hardware and software
> become blurred.

Thank you very much for the above description - very interesting.
I do however note that multiple CPUs is not a replacement for 
multi-threading (in fact multi-threading takes advantage of multiple
CPUs very well).

When you add operating system (software) to your design of multi-cpu 
system on one chip substructure it looks like:

       ASIC/FPGA
----------------------
|      |       |     |
CPU1  CPU2   CPU3   CPU4
|      |       |     |
OS    OS       OS    OS
|      |       |     |
Pr1    Pr2     Pr3   Pr4   (Pr is short for Process)

Which means the OS is replicated. In fact its even worse as the OS would 
have to be independent and use separate memory of fixed size.

In actuality when you do have multiple CPUs you still want the design like:

       ASIC/FPGA
----------------------
|      |       |     |
CPU1  CPU2   CPU3   CPU4
|      |       |     |
----------------------
          OS
----------------------
|                    |
process1--         process2--
|        |         |        |
thread1  thread2   thread3  thread4

In above typical system design the OS is shared code for entire system 
that controls what actual code is run on what CPU (and the OS functions
are not replicated for each CPU/system) and the system can share physical
memory easily. Each process is independent piece of code where as thread
is multiple instances of the same process run simultaneously to take
care of particular operation (the separation into process and thread is
so that the actual programming code is not replicated in memory for
each instance of process).

And this system would still take advantage of multiple CPUs properly 
because OS make it so that each thread actually runs on its different
CPU in parallel - but memory management and programing code is shared
and not replicated as when you have fully independent parallel systems.

-- 
William Leibzon
Elan Networks
william at elan.net