Googlebot and faulty DNS caching
Aug 23, 2002 | Q

For the first time, I’m annoyed at Google.

Moving this site from Manila to Movable Type meant moving it to another machine, which meant that the IP address for q.queso.com needed to change. There’s a way to handle the address change gracefully, and help web clients (browsers, indexers) find your new machine quickly — for the technically-minded, a few days before the move, you set the time-to-live of the nameserver entry to a shorter duration so that nameservers are sure to grab the changed entry as soon as you make it. Alas, it turns out that Google’s indexing system doesn’t play by these well-established rules. As a result, when a site changes IP addresses, Google continues to try to index at the old address, and misses any changes to the site for anywhere from a few weeks to a few months. And what I’m seeing here is that the machine running my old site continues to get hits from Google’s indexing spider, while the real updates here on the new machine are ignored. Bleah.

Comments

Exactly the same thing happened to me. If it is any comfort they eventually caught up with the new blog - a month later!

• Posted by: Euan on Aug 24, 2002, 1:33 PM

can you make the old site redirect or refresh to this locale?

• Posted by: xian on Aug 25, 2002, 3:37 PM

There’s no way to redirect it, since it’s a matter of the DNS entry on Google’s internal DNS server being wrong. So I can’t tell it to go to q.queso.com — it already is going to q.queso.com, but the Google DNS server resolves that to the wrong IP address.

• Posted by: Jason on Aug 25, 2002, 5:31 PM

When I last had this problem (campus DNS servers were all out of whack), we just internally proxied requests from machine OLD to machine NEW using IP addresses (using apache’s mod_proxy).

So folks who ended up at the old machine still saw new content, and it didn’t matter where their DNS would send them. Eventually, when their DNS caught the updated information, they’d naturally show up at machine NEW.

• Posted by: Roger on Aug 25, 2002, 10:10 PM

Google’s smart on the other end, though — they can even alias tilde accounts to domains they’ve migrated to. That’s what happened when I changed from wwa.com/~dhartung to lakefx.nu.

One can see how this can get ahead of the game in other ways, but for a search engine, that’s pretty slick.

• Posted by: Dan Hartung on Aug 26, 2002, 8:02 PM
Please note that comments automatically close after 60 days; the comment spammers love to use the older, rarely-viewed pages to work their magic. If comments are closed and you want to let me know something, feel free to use the contact page!