Activating Plan C after a Hardware Failure

Life is a crazy ride, especially when several things go wrong one after the other. When I was recently on my way back from a business trip, a thunderstorm over Britain resulted in my flight back home to Cologne being canceled. And as that was not bad enough, my cloud system at home failed catastrophically just when the flight cancellation was announced and I had a few other things to do than to deal with that.

Into Chaos

My return trip started started well enough on Friday. I arrived at Stansted airport with plenty of time to spare and I was in high hopes to be back home in Cologne at around 11 pm. As I’ve come to expect lately, my flight was at first delayed by the obligatory hour or two until it was finally canceled due to bad weather. To make matters worse, this wasn’t the only flight canceled so instead the airport getting emptier as the evening progressed, more and more people started to become stuck and the already slow LTE networks at the airport became even slower. UK network operators be ashamed!!! And just at that time I noticed that my servers at home became unreachable. Perfect timing!

Work the Problem

But first things first. As in such cases there is no point in trying to get help at the airport in finding a new flight, I went online to find another flight. The slow LTE networks certainly didn’t help but with some patience (that was hard to come by) I managed to get a flight for an exorbitant price but only two days later. No seat in a plane, no seat in a Eurostar train off the island for two days and a network at home that has just stopped responding. While my cloud services did not respond, I could at least still reach my GSM enabled power socket so I knew that power was still on and the house still standing. How glad I was once again I had this little piece of hardware in place. I couldn’t dig deeper however, because I spent the next four hours until 3 o’clock in the morning looking for my suitcase. Thanks very much Stansted airport for being so organized! But I consider myself lucky because word of mouth spread through the baggage claim hall were suitcases returned from various flights that never took off could be found. And I was by far not the last person to get his suitcase back.

Sleep Is More Important Than the Cloud

After another hour on a bus to central London and knocking on a few hotel doors until I found one that still had a room available, I finally got to bed at 5 am in the morning. Still, I had no time to get my cloud service back up and running, a few hours of sleep were more important. Finally, at 11 am with a flight ticket for the next day secured while still at Stansted and a hotel for the next night booked as well, I could focus on getting my cloud services back up and running again. Still I couldn’t reach anything from remote, neither over DSL nor over the LTE backup link. As the GSM power socket still responded, that could only mean two things: Either the UPS battery backup-up device failed or the USB switchable power strip I use to reset the backup line when it fails must have gone dead. The later would usually not have been a problem at all as the cloud server is plugged-in directly into the router. Unfortunately I was already running on plan B due to some DSL trouble and consciously decided to plug the Ethernet link of cloud server into the Ethernet hub that is also switched over the power socket before leaving in order not to be stranded if the DSL router failed. So there was no way of telling what exactly had gone wrong.

To The Rescue – Say Hello To The Backup Site

So it was time to activate plan C, which was to change the DNS entries of my domain names to point towards the backup servers at my backup site. The servers there are synchronized once a night to my main servers at home so I lost about one day worth of changes in my online calendar and some files I uploaded earlier. A small price to pay to get connectivity back instead of having to wait another one and a half days before I could potentially get back home.

Homecoming

Finally on Sunday, I was on my way back and I was obviously more than keen to rush home to see what was going on. It turned out that it was, as anticipated, the USB controlled power strip that had given up the ghost. The electronics was still working but the relays that control the individual sockets did not trigger anymore and were all off. The cloud server was still running, the DSL router was still running but everything else was off and so the server could not talk to the router, thanks to the cabling change I made just before leaving home. Unbelievable.

Hätte hätte Fahradkette (I can’t think of a good translation)

In retrospect it’s bizarre how many things had gone wrong at the same time and finally forced my cloud offline for a good 12 hours. If I had arrived in Cologne at the scheduled time on Friday, the power socked would have died only 10 minutes before my plane had arrived, so the outage would have been limited to an hour or two at most as I would have rushed home instead of trying to fix it remotely. If things had been a bit more organized at Stansted, my cloud outage time would have probably been about the same length as I would have had enough time to switch to my backup servers at another location while waiting for my suitcase. Or, better yet, there would not even have been a noticeable outage at all if I hadn’t been already running on plan B due to my DSL trouble. I would have noticed that the backup connectivity was down only after having come home.

An incredible story only real life can write…