Outage Affecting –Internet customers, Eagle Mountain, Utah exchange area.

Dec 03, 9am – We are trying to catch up and respond this morning to all the messages and calls we received yesterday. However, due to large volume of calls and emails, they might not be returned for 24 or more hours. For example, we received over 8000 comments and messages through our Facebook page. Our call center is currently working through a lot of calls from customers who factory reset their routers.

Dec 02 9:20pm – Update from engineering– New aggregation switch is completely rebuilt and the network is functioning well. Internet traffic in Eagle Mountain is back to normal. Bandwidth usage patterns all look normal. Our office phone system is back up and working too. If you are experiencing any lingering issues, please call our regular support number at 801 789 4999.

7:55pm – Update: We are pretty close to being complete on the Internet re-provisioning. Our engineers are feeling confident that almost everyone should be back up. We still don’t have our own office phone system in Eagle Mountain back up though. So, if you are still down and need to call tech support, the way to reach the Directcom Pocatello support center directly is to call 208.548.2683. This is a temporary solution during this crisis.

4:50pm – What an awful day this has been for all of you without home Internet service. It’s been a tough day for us too. We have set records for engagement on our Facebook page, and on our website.
The latest update from engineering is that they are still working hard, checking through systems, plugging away at trying to get everyone back online.

If you work from home and need an official letter to send to employers explaining that there was an Internet outage in Eagle Mountain today, please use the letter embedded below.

We know there are still some isolated customers out there still down. Thanks for your patience. We are working hard. Our techs will not go home until the network is back up and running according to his very high standards, and if you met our network administrator you would agree that he is a perfectionist when it comes to the network.

At the moment our front office phone systems are still offline while the techs focus on the priority of restoring all customer Internet service. We do thankfully have Internet service again at our front office.

Update – 12pm – Direct Communications experienced a major network outage affecting Internet customers in Eagle Mountain this morning, Dec 2, 2020. This outage appears to have began around 2am this morning due to the physical failure of a key network switch in the network central office. We followed emergency backup procedures and spent the early hours of the morning installing a backup switch. The process of rebuilding this central switch was originally projected to last until about 12pm. Our network engineers have spent many hours working on provisioning and moving physical links to the backup switch.

Internet service for the majority of customers was reported to be restored at around 11:45am.  However, our network engineers are still working on cleanup and combing through everything, so we know there may be some individual customers or subdivisions still having difficulties. If you are still not back up, please be patient–we are working through all areas as fast as we can. Please also try to report this to tech support.

We apologize for not being able to respond to your calls, messages or emails this morning. Unfortunately, our front office staff was also affected by the outage and we had no Internet or phone service at our office in Eagle Mountain. This was an unprecedented situation. Even our website crashed briefly due to the amount of traffic we received around 8am when customers tried to reach us.   We did our best to respond to customers on Facebook, but we had over 18,954 visitors on our Facebook page, and have not been able to respond to all comments there yet.

At the moment our front office phone systems are still offline while the techs focus on the priority of restoring all customer Internet service.

We did have several customers try to walk into our office here in Eagle Mountain this morning. Please also understand our front office is currently closed to foot traffic due to Covid protocols, and we are not set up to receive in-person visits to the office. Our staff cannot open the door to customers demanding a face-to-face visit at this time. Please do not try to walk into our office. Visit our Facebook page or call tech support. Any new information we receive from engineering will be posted on our Facebook page.

We do apologize for the downtime caused by this key equipment failure. This has been an extremely difficult outage. Thank you for your continued support. Please visit our website or Facebook page for outage updates at facebook.com/directcom.eaglemtn

Do not factory reset your router.

We understand that especially when so many of our customers are working from home, any downtime is a serious issue. We will be conducting a review of the systems and procedures that failed. Unfortunately, no system is perfect, despite our best planning and efforts. If you need an official letter from your ISP to send to your employer as proof of an outage, please private message us on Facebook or email our customer service staff at info@directcom.com

Although you will likely have experienced a lot of frustration today during the outage, please keep in mind that this is the first time in many, many years that we have had a full area-wide network outage in Eagle Mountain.

We really appreciate your business. Thank you for supporting Eagle Mountain’s local fiber provider.

Afterword:

I know you are probably frustrated after the events of today. However, I would ask you to consider a few points of comparison here. Some people have directed some pretty harsh criticisms at our little local company during the course of the day. I would ask the residents to check our annual filings with the Utah public service commission to see how much we have invested in the network each year if they have any doubts about our commitment to investing in Eagle Mountain. We have been building out the fiber network in Eagle Mountain for 15 years, and every year have invested several million dollars into the network. Industry professionals will agree that Eagle Mountain likely has the most modern, most advanced, most robust, most flexible, state-of-the-art fiber network in the state and there is nothing quite comparable in any large city in the state of Utah.

Every home in Eagle Mountain has fiber to the home, and the citizens didn’t have to raise a bond, or put a lien on their houses, or even pay anything extra to have that new fiber network constructed. All Internet subscription dollars go directly back into investment into the local company and local network here. We are always upgrading bandwidth and equipment. I would encourage all residents to compare notes with their family or friends living in any other city in Utah, especially those served by one of the bigger national ISPs, to see how their Internet speeds, pricing, value, reliability, and the number of outages in Eagle Mountain over the past few years, compares with their relatives’ experiences in other cities. I have a brother-in-law who’s Internet service was down for 3 days in one stretch in another city in Utah County this Fall. In several cities in our great state, the old DSL service still doesn’t go above 1.5 Mbps.

We aim to be up 99.9999% of the time, but today was just one of those days when something went really wrong, and we fell short of our goal and the expectations of our customers here, who are used to getting very reliable fiber service. We apologize for that failure. We will constantly strive to be better.