How VTel’s phone/internet outage happened

By Shawn Cunningham
© 2020 Telegraph Publishing LLC

Multiple equipment failures brought to a standstill VTel phone and internet service to Chester, Andover, Grafton and Windham — first on Thursday, then on Friday and into Saturday of last week, prompting questions about access to public safety agencies and how to make that more robust.

According to a detailed timeline of the “network event” provided to The Telegraph by VTel, the issues began on Thursday midday when technicians became aware of a problem communicating with Chester, which at first looked like a cut somewhere in the system’s fiber optic cables. None were found, but it appeared to technicians in Springfield that traffic was not passing through a router in Chester.

Within 15 minutes, according to the timeline, two technicians and two engineers arrived and began diagnosing the cause of the outage. That work was hampered — ironically — by the lack of internet service to verify equipment settings. An hour or so of working on getting the router to respond resulted in finding that the device’s configuration was blank.

VTel Chief Technology Officer Arianna Robinson noted that experience has shown that uploading a new configuration is more effective if done in batches, so engineers began the hour-long task of uploading 3,905 lines of code. By late Thursday afternoon, the controller cards in the router rebooted and began to provide service. After taking care of a few other problems, service was back around 5 p.m.

But, early Friday morning the router again failed and, after troubleshooting the problem, the installation of a new router was begun. At the same time, engineers tried to bring the old router back online whenever it crashed to provide some service.

With new hardware installed, “multiple engineers” working onsite and remotely, migrated services to the new router.  This was done slowly as reprogramming was also being done at the same time. According to Robinson, the process took until the early evening when another crash occurred and the process began over again with engineers resolving problems as they went. After the situation seemed to be resolved and had been watched for an hour, the crews headed home around 11 p.m. Friday.

On Saturday engineers did some de-bugging to solve some left-over problems with video and internet that a few customers were having and service was returned to normal.

Phone company addresses public safety concerns

The lack of phone service meant that customers within the affected area were unable to access 911 services, a situation that has been a topic of conversation among many customers who wonder how a repeat of the outage could be avoided.

“This outage was an eye-opening experience, and one that has led to plans to ensure we don’t see a repeat of this widespread occurrence,” said Amber Wilson, VTel’s director of Customer Care, who told The Telegraph that one example of changes under development includes a division of functions within VTel’s network design.

Wilson explained that the company is looking to separate larger processes that occur on its network on a daily basis into more compartmentalized pieces. “That way, any failure or errors on one component will affect fewer customers and lead to quicker restoration.”

“This and other system changes are critical for our network,” said Wilson, who noted that while VTel does not have a reputation for outages, if another unexpected outage were to occur for any reason, the company’s  goal will be to deliver a much quicker, much smoother recovery. “The reliability of our network has always been a priority and will continue to be a priority, during the pandemic and beyond,” she said, adding that that VTel’s commitment to customers “has never been more important as the world’s needs have seemingly changed overnight.”