Country Cablevision Outage Report.
Title: May 11th Wake County Outage Report
May 16, 2020
By Cablevision Team.

During the week, to be precise, Monday 11th of May, 2020, we experienced failure in one of our cable lines supplying wake county failed. As a result, from 10:30 - 12:45, internet service wasn't provided to the wake county. Today, we present an incident report that details the happenings and the reason for the outage, with our detailed response to it.
Country Cablevision Outage Report
The staff was notified around 10:35, and our response team immediately went to wake county to deal with the situation. The RCA was done on the causes of the outage. According to it, it was shown that a certain network connectivity hardware model failed, and had not been changed like the others used within other counties. This hardware had been known to fail, and therefore, the outage could have been prevented

Timeline

10: 30: Occurrence of outage
10: 35: Staff was alerted
10: 56: Team got to the network provision site area
11: 24: Identification of Issue
11: 37: Successful fixing of the new hardware
12: 10: Servers are full back online (for Wake County)
12: 45: 100% of users have access to internet

Root Cause Analysis

A particular networking hardware – dense wavelength division multiplexers (DWBM) which had been changed at different site was unfortunately not changed at the wake county during the usual scheduled maintenance. This exposed the network setup to lower connectivity, network congestion and also, what occurred, system failure. This failure caused our entire Wake county to be blocked from the servers and inherently from the internet. The cause of the failure of theDWBM, however, is a semiconductor within the old networking model, and due to this service outage occurred.

Resolving and Recovering

Certain users notified our team via the customer care over lack of internet connectivity. As the team was working on another county area because of the recently scheduled maintenance and testing that was going on at the time, we didn't notice the issue on time, and for that we are very sorry. On being escalated by the customer care, the teams’ engineers noticed and did a quick assessment of the situation leading them to identify where the problem could be at. On getting to the site, troubleshooting started and several forced connections was made to the main server which failed even though our test showed everything seemed fine. However, on a quick inspection of the all hardware, it was noticed that the proxy server still in use was the old ones. It was immediately replaced and the connections was forced to the main, and the time around it was successful. The sub servers at Wake were restarted gradually in order to avoid another shutdown. By 12: 45 PM, the issue was resolved and all of wake county were back online

Resolutions

Over the last few days, we've conducted an analysis and measure taking review of the outage, and certain actions were identified to be taken to address the inherent cause and probable causes of this issue and other related issues. Also, it was noticed that it took a lot of time to come back online which definitely don't want happening again. First, we don't even want an outage, but in the case where an outage is inevitable, our comeback time should be short. The actions below where agreed upon;

1. All shifts are required to do continuous hardware check on all the systems instead of the daily check-ups which were in place.

2. After every scheduled maintenance, a thorough cross-check of hardware must be done via a checklist - containing the information of the hardware that are due to have been changed.

3. Improve our comeback time by involve a less number of people on certain shutdown as too many engineers slow down the process.

At Cablevision, we are committed to giving the great people of North Carolina, stable and continuous internet land internet supply for their use at home and at the workplace. We are absolutely sorry for the inconveniences caused. We know that no amount of apology can erase the impact the shutdown had on you and your businesses. However, we are very sorry and do promise to give you the best service. Thanks for your patience and continued support. We are only here but you want us to be here. Thanks once again.

Our warmest regard,
Head of Networking,
Country Cablevision.