That’s it - today is the day the internet broke! People started realising something was wrong with Google when almost every search result had the accompanying text “This site may harm your computer”. Clicking on any result led to a page warning the user that the page they wanted to visit was very likely a page which would try to do bad things to your computer. Thankfully, the glitch only lasted for 30-45 minutes, and things were back to normal again, with an explanation of what went wrong. And the world breathed a sigh of relief… or did they?

A few minutes ago, I got an e-mail alert from Hyperspin. Hyperspin monitor your servers, and e-mail you if something goes wrong. Apparently something had - it was failing to resolve my domain name! A few nslookups showed that this really was the case. Worse - none of my domains were being resolved. I try to access the eNom website, and find that even THEIR website isn’t working.

Ok, fine - so something’s wrong with eNom’s DNS servers - all 5 of their geographically-separated locations! None of them respond! A global failure of their DNS servers for more than a few minutes is really unthinkable. They say on their web-site:

eNom services set new standards for reliability, thanks in part to redundant name servers dispersed around world. Each server has multiple high-bandwidth Internet connections, back-up power, security, and access to three different major Internet backbones. This powerful system enables to handle millions of transactions with no interruption in service.

I thought I’d call their tech-support to see what they have to say. Of course, their number is listed on their web-site, and that’s completely unreachable right now. Thankfully, the web archive was helpful to locate and find an archived contact page from their site. So I call up the number listed, dutifully press “3″ for technical support and hear the message “Please enter your support PIN - this PIN is available within the ‘Info’ section of the ‘My Account’ page on our site”. AAARGH! Ok … don’t panic. A quick visit to Netcraft, and I’m in possession of the IP address they last switched to. http://69.64.157.35 does the trick, and I’m greeted with the familiar Enom home page. Login, get my phone-support PIN and call them again. I’m not that surprised that I get through to a support drone almost immediately - most of their customers are still battling their way throgh to their phone support PIN! Unfortunately, the support drone was of no help at all. He acknowledged that there was a problem, that their entire team of engineers was investigating what had gone wrong, but no, they don’t have an ETA as to when things will be sorted out.

It’s been over half an hour since I got the initial alert mail, and nothing has changed yet. Wonder how long this will take to fix.

Update:

1.5 hours later, it looks as if things are recovering. Most queries do get a valid response, although some of them do still time-out.

Of course, their Outage reporting site was also completely inaccessible. And now, even when you are able to access it, all it says is “Unscheduled Maintenance - Our site is currently undergoing an unscheduled maintenance to upgrade our systems in order to better serve you.”

Final update:
I just received a response to the ticket I filed with eNom. Turns out the entire thing was due to a DDoS attack.

Hello,

Thank you for contacting us regarding the recent site resolution issues you were experiencing.

For a period of hours beginning a Noon PST on Saturday January 31, 2009, a eNom DNS servers were victim to a large Distributed Denial of Service (DDoS) attack. This attack affected hosted customers and other services, which rely on our DNS infrastructure. Our DNS regularly handles attack traffic during the normal course of business; however, this attack was particularly large and required additional effort by eNom Operations to counteract it. Services were largely restored by 3:30pm PST. By policy, eNom doesn’t detail the nature of attacks against our infrastructure.

Thank you for your understanding and patience.

Technical Support,

It’s difficult to identify a single page or site on the Net as being yours. Sure, you may have your own blog or even a completely independent website, but with sites like Del.icio.us, Flickr, FriendFeed and your profiles on all these pages, you’re bound to have at least a half dozen other pages that are “yours”. So now that you’re on these services, what should you do? To get the most of them, you’ll of course need to find your buddies on these sites, and that’s where things get tough. How do you know that the John Smith on service XYZ is in fact the same John Smith that you play scrabble with? If John has linked from his web page to his user profile then it’s no problem, but what if he’s only got a link the other way around, from his user-profile page back to his web-site. You shouldn’t have to click through the dozens of profile pages to find his - let Google do the work for you through the Google Social API!

A few months ago, Google made available an API which you can use to query the social network inherent in the web, by following the “me” links or pointers from one network’s profille to another, as well as the “friend” pointers. By then giving the URL of my web site, I can find all my “other” pages on the Internet, as well as everyone else that I link to or am linked by.

Using this social graph of services, it’s easy for you to find what services you have in common with someone else to be able to link with them there. To make this easier I have a very trivial bookmarklet I can invoke when I’m on any site, which does a lookup via Google Social’s API. To use it, simply drag and drop the “Google Social API Lookup” link to your bookmarks toolbar. Click it when you’re on any blog or profile page, and you’ll do a Google Social lookup. Enjoy!

A lot of people know that I’ve been working with Selenium to automate UI tests. You can now see a few examples of how specifically we use Selenium at Google, via the publicly released Selenium-based tests used as part of the Maps API testing process.

When you’re developing a Web-based application, testing a Web UI can get quite time-consuming. Throw in the problem of having to support multiple browsers, and the complexity of AJAX, and things can get out of hand quite quickly. This is where having an automated set of UI tests can be a life-saver. Make these tests part of your CI process, and you’ll get feedback as soon as something is broken.

Often, people end up testing the UI as part of an end-to-end test. This isn’t necessarily the best way to be testing the UI, as you’re no longer able to pinpoint the location of problems to a particular layer. Is that problem you’re seeing a data-encoding issue of the HTTP request/response, the database, or the persistence API? It could be anywhere. So take advantage of that multi-tiered system design which you have. Test each layer in isolation, separate it from the other layers that it’s talking to by putting fake, mock or stub layers in place, and test that layer by pumping in events and calling methods. Check the calls appearing out of that layer to verify they’re what you expect, and you have a much more useful set of tests.