That’s it – today is the day the internet broke! People started realising something was wrong with Google when almost every search result had the accompanying text “This site may harm your computer”. Clicking on any result led to a page warning the user that the page they wanted to visit was very likely a page which would try to do bad things to your computer. Thankfully, the glitch only lasted for 30-45 minutes, and things were back to normal again, with an explanation of what went wrong. And the world breathed a sigh of relief… or did they?
A few minutes ago, I got an e-mail alert from Hyperspin. Hyperspin monitor your servers, and e-mail you if something goes wrong. Apparently something had – it was failing to resolve my domain name! A few nslookups showed that this really was the case. Worse – none of my domains were being resolved. I try to access the eNom website, and find that even THEIR website isn’t working.
Ok, fine – so something’s wrong with eNom’s DNS servers – all 5 of their geographically-separated locations! None of them respond! A global failure of their DNS servers for more than a few minutes is really unthinkable. They say on their web-site:
eNom services set new standards for reliability, thanks in part to redundant name servers dispersed around world. Each server has multiple high-bandwidth Internet connections, back-up power, security, and access to three different major Internet backbones. This powerful system enables to handle millions of transactions with no interruption in service.
I thought I’d call their tech-support to see what they have to say. Of course, their number is listed on their web-site, and that’s completely unreachable right now. Thankfully, the web archive was helpful to locate and find an archived contact page from their site. So I call up the number listed, dutifully press “3” for technical support and hear the message “Please enter your support PIN – this PIN is available within the ‘Info’ section of the ‘My Account’ page on our site”. AAARGH! Ok … don’t panic. A quick visit to Netcraft, and I’m in possession of the IP address they last switched to. http://22.214.171.124 does the trick, and I’m greeted with the familiar Enom home page. Login, get my phone-support PIN and call them again. I’m not that surprised that I get through to a support drone almost immediately – most of their customers are still battling their way throgh to their phone support PIN! Unfortunately, the support drone was of no help at all. He acknowledged that there was a problem, that their entire team of engineers was investigating what had gone wrong, but no, they don’t have an ETA as to when things will be sorted out.
It’s been over half an hour since I got the initial alert mail, and nothing has changed yet. Wonder how long this will take to fix.
1.5 hours later, it looks as if things are recovering. Most queries do get a valid response, although some of them do still time-out.
Of course, their Outage reporting site was also completely inaccessible. And now, even when you are able to access it, all it says is “Unscheduled Maintenance – Our site is currently undergoing an unscheduled maintenance to upgrade our systems in order to better serve you.”
I just received a response to the ticket I filed with eNom. Turns out the entire thing was due to a DDoS attack.
Thank you for contacting us regarding the recent site resolution issues you were experiencing.
For a period of hours beginning a Noon PST on Saturday January 31, 2009, a eNom DNS servers were victim to a large Distributed Denial of Service (DDoS) attack. This attack affected hosted customers and other services, which rely on our DNS infrastructure. Our DNS regularly handles attack traffic during the normal course of business; however, this attack was particularly large and required additional effort by eNom Operations to counteract it. Services were largely restored by 3:30pm PST. By policy, eNom doesn’t detail the nature of attacks against our infrastructure.
Thank you for your understanding and patience.