SNMP - monitoring large number of devices

Tue Sep 29 21:03:18 UTC 2015

Thus spake Dan White (dwhite at olp.net) on Tue, Sep 29, 2015 at 03:37:51PM -0500:
> On 09/29/15 22:20 +0200, Pavel Dimow wrote:
> >recently I have been tasked with a NMS project. The idea is to pool about
> >20 OID's from 50k cable modems in less then 5 minutes (yes, I know it's a
> >one million OID's). Before you say check out some very professional and
> >expensive solutions I would like to know are there any alternatives like
> >open source "snmp framework"? To be more descriptive many of you knows how
> >big is the mess with snmp on cable modem. You always first perform snmp
> >walk in order to discover interfaces and then read the values for those
> >interfaces. As cable modem can bundle more DS channels, one time you can
> >have one and other time you can have N+1 DS channels = interfaces. All in
> >all I don't believe that there is something perfect out there when it comes
> >to tracking huge number of cable modems so I would like to know is there
> >any "snmp framework" that can be exteded and how did you (or would you)
> >solve this problem.
> 
> I've done about ~60,000 OID queries (over a few dozen devices) per 5
> minutes using OpenNMS, which is Java based. At the scale you're looking at,
> disk I/O would be a major performance issue (if using rrdtool). Google for
> 'Tuning RRD' for some tips that can make a significant difference.

These days all you need is SSD.  We have about 800,000 active rrd files
(in MRTG, for that matter) on one old server.

Otherwise the various techniques we proposed in 2007 are now largely
implemented, particularly rrdcached beginning in version 1.4.
https://www.usenix.org/legacy/event/lisa07/tech/full_papers/plonka/plonka_html/index.html

Dale