Country
Cablevision Outage Report.
Title: May 11th Wake County Outage Report
May 16, 2020
By Cablevision Team.
During the week, to be precise, Monday 11th
of May, 2020, we experienced failure in one of our cable lines supplying wake
county failed. As a result, from 10:30 - 12:45, internet service wasn't
provided to the wake county. Today, we present an incident report that details
the happenings and the reason for the outage, with our detailed response to it.
The staff was notified around 10:35, and
our response team immediately went to wake county to deal with the situation.
The RCA was done on the causes of the outage. According to it, it was shown
that a certain network connectivity hardware model failed, and had not been
changed like the others used within other counties. This hardware had been known
to fail, and therefore, the outage could have been prevented
Timeline
10: 30: Occurrence of outage
10: 35: Staff was alerted
10: 56: Team got to the network provision
site area
11: 24: Identification of Issue
11: 37: Successful fixing of the new
hardware
12: 10: Servers are full back online (for
Wake County)
12: 45: 100% of users have access to
internet
Root Cause Analysis
A particular networking hardware – dense
wavelength division multiplexers (DWBM) which had been changed at different
site was unfortunately not changed at the wake county during the usual
scheduled maintenance. This exposed the network setup to lower connectivity,
network congestion and also, what occurred, system failure. This failure caused
our entire Wake county to be blocked from the servers and inherently from the
internet. The cause of the failure of theDWBM, however,
is a semiconductor within the old networking model, and due to this service
outage occurred.
Resolving and Recovering
Certain users notified our team via the
customer care over lack of internet connectivity. As the team was working on
another county area because of the recently scheduled maintenance and testing
that was going on at the time, we didn't notice the issue on time, and for that
we are very sorry. On being escalated by the customer care, the teams’
engineers noticed and did a quick assessment of the situation leading them to
identify where the problem could be at. On getting to the site, troubleshooting
started and several forced connections was made to the main server which failed
even though our test showed everything seemed fine. However, on a quick
inspection of the all hardware, it was noticed that the proxy server still in
use was the old ones. It was immediately replaced and the connections was
forced to the main, and the time around it was successful. The sub servers at
Wake were restarted gradually in order to avoid another shutdown. By 12: 45 PM,
the issue was resolved and all of wake county were back online
Resolutions
Over the last few days, we've conducted an
analysis and measure taking review of the outage, and certain actions were
identified to be taken to address the inherent cause and probable causes of
this issue and other related issues. Also, it was noticed that it took a lot of
time to come back online which definitely don't want happening again. First, we
don't even want an outage, but in the case where an outage is inevitable, our
comeback time should be short. The actions below where agreed upon;
1. All shifts are required to do continuous
hardware check on all the systems instead of the daily check-ups which were in
place.
2. After every scheduled maintenance, a
thorough cross-check of hardware must be done via a checklist - containing the
information of the hardware that are due to have been changed.
3. Improve our comeback time by involve a
less number of people on certain shutdown as too many engineers slow down the
process.
At Cablevision, we are committed to giving
the great people of North Carolina, stable and continuous internet land
internet supply for their use at home and at the workplace. We are absolutely
sorry for the inconveniences caused. We know that no amount of apology can
erase the impact the shutdown had on you and your businesses. However, we are
very sorry and do promise to give you the best service. Thanks for your
patience and continued support. We are only here but you want us to be here.
Thanks once again.
Our warmest regard,
Head of Networking,
Country Cablevision.
0 Comments