Enhancing automation with network growth

Thu Jan 21 03:52:39 UTC 2010

> 
> I'm reaching the point where adding in a new piece of infrastructure
> hardware, connecting up a new cable, and/or assigning address 
> space to a
> client is nearly 50% documentation and 50% technical.
> 
A common problem :)

> One thing that would take a major load off would be if my MRTG system
> could simply update its config/index files for itself, instead of me
> having to  do it on each and every port change.
> 
> Can anyone offer up ideas on how you manage any automation in this
> regard for their infrastructure gear traffic graphs? 
> (Commercial options
> welcome, off-list, but we're as small as our budget is).
> 
Not sure how you're doing your graphs currently, but have you considered Cacti?

> I use a mix of Cisco/FreeBSD&Quagga for routers, and Cisco/HP for
> switches, so it is not as simple as throwing a single command at all
> configs.
> 
> All feedback welcome, especially if you are in the same boat. My IP
> address documentation/DNS is far more important than my traffic stats,
> but it really hurts when you've forgotten about a port three 
> months ago
> that you need to know about now.
> 
First, I'll throw out a bit of what we do and it might give some ideas, though not necessarily good ones. We use Linux/Quagga routers, in-house-modified Linux-based LNSs, and HP switches. Some of our configuration and change management is done via cfengine, backed by subversion. The latter yields the added benefit of revision control and all the other good stuff you can get from svn in such a scenario. Unfortunately this is only part of the config/graphs/docs/DNS/IPs/OSS equation and we don't have everything fully integrated yet (nor is there a business case for it at the moment). Some of our OSS is based on a heavily in-house modified version of Freeside as well as our own app/layer that sits on top. This is essentially our base system which allows us to push data and prov services to other internal and external systems - e.g. DNS, IP assignment, vendor's portals/APIs, RADIUS, etc. (basically almost every piece of hardware and software we have) and which interfaces with our self-service (customer portal - aka the almighty call-avoidance "solution"). We also use IPPlan for managing IP assignment, but are moving away from it.

In a perfect world, everything would be integrated with everything else, searching by every data element would be possible, every business process would be automated, all of your docs would be in one place, all linked to the network element / customer / ticket / order / whatever, and so on. For most organizations, this is neither feasible nor required. Each system tends to do one or two things well and you have much unavoidable data duplication and data moving back and forth. Usually the goal is to minimize the amount of manual data entry down to a single time and to push this aspect out towards the customer as much as possible. The extent of that will depend on your specific environment - everyone basically does the same thing, so often there's no need to re-invent the wheel (i.e. "let's develop everything from scratch in-house" is a very bad move - you're not in the OSS business).

OSS/BSS is a huge and complex topic, so I'm only touching the tip of the iceberg here and speaking mostly in general terms. It's definitely something that will be of greater and greater importance as your network grows, so early planning is key, but don't get carried away trying to automate the hell out of everything because you'll lose focus on what you need to do in the short-term.

There is often a naive pursuit of perfection in OSS/BSS by those who haven't been doing it for long enough - don't fall into that trap.

I'd start by defining your requirements/scope more solidly and then considering whether it makes sense to try to automate or enhance a particular process. It may help to break things down step-by-step, perhaps based on dependencies or some other logical order, then think about how you would eliminate what you perceive to be manual/error-prone/inefficient/slow/whatever. From a costing perspective, you might find yourself in a (unfortunately frequently encountered by some) situation of "I just spent 50 hours writing a program to automate a task that would have taken me 2 hours to do manually" or "we just spent $50k buying a product which we won't use to any reasonable level of capacity for the next five years".

--
Erik
*** Remove the _list part in my e-mail address to reply. ***