Server outage

For the past twelve years my domain has been hosted by a regional company (separate from my ISP), but although problems have been few and far between (and were always solved swiftly), I was forced to take over and handle it myself.

Here's a record of my ordeal:

Tue 2013-02-05, day one

At 09:15 UTC the lights suddently went out at my hoster's data center (but I didn't notice until the next day).

My logs show that my local server successfully fetched a batch of e-mails at 09:13 UTC, but two minutes later fetchmail stopped in the middle of the next batch.

Since then, all connection attempts timed out.

Wed 2013-02-06, day two

I wondered why I hadn't received any e-mails (not even the occational SPAM) for the past day and took a look at my server logs.

Having noticed the timeouts I tried to contact the hosting company, but their land lines (VoIP) were dead. I couldn't reach their home-page either, so I wondered if perhaps their backbone was damaged.

Not knowing what was going on or even if the problem was transient or not, I patiently waited.

Thu 2013-02-07, day three

Still no connectivity to the hoster and no news whatsoever about the incident, so I started preparing for the worst.

As the hoster was located in my region (less than 20 km away) I decided to drive by after work and have a look at the surroundings.

I got there about half an hour before the end of their office hours, but I could not spot neither light nor any other sign of life in the entire building, so I decided to act.

At the advice of a knowledgeable collegue I opened an account with the domain registration robot of the German hoster Hetzner Online AG in order to move my domain elsewhere.

I placed the order rather late in the evening, so I just had to wait until the next day.

Fri 2013-02-08, day four

I received the access code to the domain registration robot in the morning and immediately initiated a domain transfer.

Alas, my request was rejected; it turned out I had to provide an authorisation code which the domain registrar is supposed to provide.

Now things threatened to get out of hand - the old hoster (still unreachable) was also the registrar, so I had to quickly learn how the Internet works (organisationally, that is).

My hoster was a member of CORE Internet Council of Registrars in Geneva, so I emailed them that my hoster/registrar was Missing In Action and asked for help.

The person from the CORE secretariat was friendly and helpful; after having cleared some formalities regarding proof of identity I was handed the precious authorisation code, only two hours after the first contact.

With the authorisation code at hand, the robot finally accepted the domain transfer request.

The confimation stated that the actual transfer would only take place after five days (unless the previous registrar would ACK the transaction earlier, but alas he was still unreachable).

I guess I could have expedited the transfer (by contacting CORE again), but as it was now friday afternoon and I still hadn't decided where to host my domain, I instead started searching for a new hoster.

Sat 2013-02-09, day five

I spent an hour or two looking for and comparing virtual servers and root servers, since by now I had more-or-less abandoned the possibility of selecting a new web-hoster but instead decided to regain complete control by renting a server and configuring the web- and email-services myself.

Sun 2013-02-10, day six

I decided to rent the smallest (read: cheapest) root-server from Hetzner, still a fairly powerful box (i7 quad-core, 16 GiB RAM and 2 x 3 TB disk drives), suitably for hosting a handful of virtual machines.

My plan was to compartmentalise the services (e-mail, web, whatever), each in its own VM.

Mon 2013-02-11, day seven

The server was ready, complete with a Ubuntu 12.04 minimal installation.

Having restricted access according to my wishes the next step was to install and configure OpenVPN on the server to get a comfortable routing to and from my LAN.

Next I installed qemu-kvm and prepared a 'template'-VM, also with Ubuntu 12.04.

I then cloned the template and configured my new mail-server.

Since my domain was still lingering awaiting transfer, I could only test that local emails were handled correctly, though.

Tue 2013-02-12, day eight

I configured the next VM as a web-server and uploaded the backed-up content of my home-page.

Using iptables I redirected incoming traffic on port 25 to the mail-VM and port 80 to the web-VM, then verified the connectivity.

Wed 2013-02-13, day nine

Early in the afternoon I got the longed for message that my domain was finally transferred, so I configured DNS, logged into the mail-VM and let /var/log/mail.log run by.

Less than a minute after having submitted my DNS zone file, the first e-mail arrived: SPAM...

At any rate, it meant that after nine days I was now up-and-running again.

Oh, just noticed that my new server came with a global /64 IPv6 subnet! Hmmm...