outages, quality monitoring, trouble tickets, etc

Sat Nov 25 00:18:12 UTC 1995

Sorry, another long one.

Relevance to NANOG, well, NOCs are customers too.  And "remote" NOCs
often report problems that effect your paying customers too.  Customer
service should be of interest to operations folks, at least to the extent
the problems are getting reported to the right people to fix.

I doubt I can change anyone's mind that providing explanations to
customers and non-customers when the network has problems is good for
business.  In the future I will simply recommend to customers to buy
services from NSPs which do provide explanations when their networks
fail.  Since I haven't found a perfect network yet, I suspect it
includes everyone on this list.

>  Why should they talk to you?  Do you pay them a service fee?
>  That's my base issue, there is a hierarchy, and you can't skip
>  rope to the other guy.  It just doesn't work, there's nothing in
>  the system to encourage it.

The hierarchy is dead.  None of the old NSFnet regional have a
monopoly on service in their regions any more.  Outside of the US,
there are still a few monopoly providers, but they are a rare breed. If
you aren't providing the level of service I need, I'll go to someone who
can.  If XYZ's NOC gives me better service than ABC's NOC, I'll
recommend XYZ to my customers.

>  Sean Donelan has a terribly good point, he's my customer, and his
>  words mean alot, but I can't agree w/ him that he should/could
>  demand the same thing from another ex-NSFnet regional, or from
>  Sprint.  I certainly see no reason why I should do this work for
>  you.  

Because it is in their self-interest?  You are correct I can't make
anyone run their network how I would like it run, not even MIDNET (GI).

But I can point out long-term problems and code of silence is costing such
providers money, and has already cost them customers.  For example, I really
wish my direct providers would stop munging BGP announcements, or explain
why they are doing it.  If I have made a mistake, I would like to fix it.
Otherwise I will come to the conclusion those provider's NOCs are not up
to the job and find a different provider that can do the job.

When someone (anyone) reports a problem effecting connectivity with
your network, more than likely the reverse is also true for your paying
customers.  DRA has a bunch of customers connected through just about
every major NSP in North America and a couple of other continents.  The
only time "I" call another NSP is when the process has become totally
FUBARed.  When I call another NSP, it is usually that NSP's last chance
to keep a paying customer on their network.

I might call BARRNET because the University of California-Davis has
reported problems reaching DRA to DRA's help desk, and the problem hasn't
been resolved.  No, BARNET doesn't *have* to talk to me.  And I will
report the same back to the customer.  However, I suspect it is in
BARRNET's self-interest to work with me in resolving the problem
to ensure UC-Davis has end-to-end reliability.

I track network reliability by dollars (not packet loss, not latency).
I measure network providers, good and bad, by how many of our customers
have used their own dollars to buy private lines to St. Louis because
they couldn't get the reliability they needed from the network provider.

It is not a pretty picture.  <http://dranet.dra.com/dranet.html> has
a picture where our private line customers are located.  If you are an
NSP, every one of those green boxes (some boxes represent many paying
customers) is an arrow through the heart of your (former) customers view
of your network reliability.  If you are an NSP in one of those areas,
not dealing with these problems or providing coherent explanations has
cost you cold-hard cash.

Money is something I expect most upper managers to understand.  DRA makes
its profits elsewhere.  DRAnet is simply a vertical market VPN used to sell
access to other things.  I'm happy to use the Internet and other NSP's to
provide that VPN, when the quality exists.  On the other hand, if I have
to manage a not-so virtual VPN with private lines to achieve the required
level of quality, I do.

Maybe Adam Smith's invisible hand will correct this eventually.

>  There seems to be this large obsession with linking information to
>  action.  If you get an update you think something's happening.
>  Perhaps it's needed, but stuff will happen whether your hand is
>  held or not.

As I said before:  Ideally I want a reliable network.  If you can't
provide a perfectly reliability network I want an explanation when I
can't get through.  And I want the problem fixed.  The better the
explanation, the longer I'm willing to give you to fix the problem.  If
I get no explanation, I expect the problem to already be fixed.

The current situation is the customer gets neither the explanation nor 
action solving the problem.  My proof is the DRAnet map.  DRA's customers
take a very, very long time to budget money.   Those green boxes represent
customers whose problems went unanswered, and unsolved for a long time
before they gave up on their NSP and expended their own dollars for a
private line to St. Louis.

Since the technicians seem to be having a very difficult time fixing
the network, I thought upper management could meet my other goal.  Give
the customer an explanation.  I'm not pointing fingers at any particular
NSP, because frankly I don't have enough fingers to point.  Everyone had
problems.  Yes, even DRA's NOC has fallen down a few times.  I'm not asking
for perfection, but an explanation when things don't work, while you
fix the problem.

The Internet is a global cooperative network.  If people don't cooperate,
the global nature of the network fails.  Since your customers may in fact
want to use the Internet to communicate globally, problems effect customers
globally.  When I go to the US Post Office, sometimes there is a sign on
the wall that postal service to Timbukto may be delayed because Timbukto's
main post office was blown up.   I have no idea how many postal customers
in Olivette, Missouri send mail to Timbukto.  Even though the US Post Service
has no control over rebuilding Timbukto's main post office, the US Post Service
has discovered it is good customer service to inform their customers why
their mail to Timbukto may be delayed.

Can't NSPs provide their customers an explanation at least as well as
the US Post Office?
--
Sean Donelan, Data Research Associates, Inc, St. Louis, MO
  Affiliation given for identification not representation