In all the many years that I’ve hosted websites, we’ve never had an outage this serious. Here’s what happened…
At approximately 2:30pm on Wednesday, November 1st, 2017, our server went dark with no explanation. 13 hours later at 3:30am the next morning, the server was finally brought back online.
When the server first went down, our first actions were to check to see if there was something wrong with the server. We quickly learned that the server itself was fine but that something had happened to the internet coming into the data centre.
For those who don’t know, our main server is located in an extremely fortified data centre near Ottawa, Ontario, Canada. It turns out that at about 8km away, a truck had clipped a fibre optic cable that was strung across two telephone poles which caused the problem. The only thing we could do was wait… and wait…
The fellow in charge of managing the physical server and it’s environment is named Sean. Here is his report this morning after the cable was reconnected…
When visiting the incident site in the field at about 2am this morning (a humanitarian “cheeseburger and hot coffee” run for the two dozen linesmen and network specialists), we were able to understand more about what had happened (an over-height vehicle clipped the lower sections of a utility pole circuit, pulling over a light standard and just about dropping a hydro pole onto a house), the scope of work to be done (remove, replace and splice in a new 200m section of 482 strand fibre) and the intricacies of the work (two cable engineers, at both ends of the new section, madly trimming, cleaning, splicing and sealing strand ends).
Thank you for your patience during this very unfortunate service interruption… and a big thanks goes out to my team who, despite having nothing to do, still managed to respond and react in a solid, professional manner. Had the outage been caused by anything involving our server or the data centre, I’m confident that it would have been dealt with with the same level of professionalism.
In the future, be sure to check out our twitter feed first as I always post updates there… https://twitter.com/lightspeedguru
If you still wish to communicate with me during a server outage, always go to http://kristingreen.freshdesk.com and submit a ticket there. This ticket system is not dependent upon our main server as is hosted by the folks at FreshDesk.
Here are some pictures that Sean sent over for us.
