Thursday, November 26, 2009 at 3:23 AM
How to use "Fetch as Googlebot", part 1Webmaster level: Intermediate
Has your site ever dropped suddenly from the index or disappeared mysteriously from search results? Have you ever received a notice that your site is using cloaking techniques? Unfortunately, sometimes a malicious party "hacks" a website: they penetrate the security of a site and insert undesirable content. Sophisticated attackers can camouflage this spammy or dangerous content so that it doesn't appear for normal users, and appears only to Googlebot, which could negatively impact your site in Google's results.
In such cases it used to be very difficult to detect the problem, because the site would appear normal in the eyes of the user. It may be possible that only requests with a User-agent: of Googlebot and coming from Googlebot's IP could see the hidden content. But that's over: with Fetch as Googlebot, the new Labs feature in Webmaster Tools, you can see exactly what Googlebot is seeing, and avoid any kind of cloaking problems. We'll show you how:
Let's imagine that Bob, the administrator of www.example.com, is searching for his site but he finds this instead:

That's strange, because when he looks at the source code of www.example.com, it looks fine:

With much surprise Bob may receive a notice from Google warning him that his site is not complying with Google's quality guidelines. Fortunately he has his site registered with Webmaster Tools, let's see how he can check what Googlebot sees:
First Bob logs into Webmaster Tools and selects www.example.com. The Fetch as Googlebot feature will be at the bottom of the navigation menu, in the Labs section:

The page will contain a field where you can insert the URL to fetch. It can also be left blank to fetch the homepage.

Bob can simply click Fetch and wait a few seconds. After refreshing the page, he can see the status of the fetch request. If it succeeds, he can click on the "Success" link...

...and that will show the details, with the content of the fetched page:

Aha! There's the spammy content! Now Bob can be certain that www.example.com has been hacked.
Confirming that the website has been hacked (and perhaps is still hacked) is an important step. It is, however, only the beginning. For more information, we strongly suggest getting help from your server administrator or hoster and reading our previous blog posts on the subject of hacked sites:
- My site's been hacked - now what?
- A quick security checklist for webmasters
- Message center warnings for hackable sites
- Best practices against hacking
If you have any questions about how to use the Fetch as Googlebot feature, feel free to drop by the Webmaster Help Forum. If you feel that your website might be hacked but are having problems resolving it, you might want to ask the experts in our "Malware and Hacked sites" category.
PS Keep in mind that once you have removed hacked content from your site, it will generally still take time for us to update our search results accordingly. There are a number of factors that affect crawling and indexing of your content so it's impossible to give a time-frame for that.


20 comments:
Very kewl! Thanks for coming uop with this one guys!
Thanks! Now I can sleep at night. ;)
Anyone knows a tool to monitorize this? IT could be interesant, like a SPAM filter.
For example you give to the tool the sitemap of your site and it check daily if any suspicious link have been added.
Hey,
Thanks for letting me know about this feature. I will make a note of it. Hope in future it helps.
Thanks
Sankar
I attend to know this pretty much better, thanks a lot.
Thats relly a nice info...
www.mgingerhack.blogspot.com
oh that feature is interesting, will be showing it in proper html, less confusing....
Thanks for improving the webmaster tool..
dani26381, there are tools that can do this.
Get UpdatePatrol. It will monitor your or your clients' sites and notify you when changes occur.
http://www.updatepatrol.com
Thanks for the information. Apart from Webmaster Tools do anyone know any alternate way to figure out this?
my site too has been hacked las week but i have corrected all the errors you can see at: onjeux.com
Thanks for the information
even if the site has not been hacked it can suddenly disappear from the results. Why we don't know but our site www.takethis.nl which held a 3rd postion for the keyword sexshop for a long time is nowhere to be found for this keyword. also a few of my competitors are nowhere to be found as well. If anyone can think of a good reason for this please tell us.
Hi to all. I think can be good to use the fetched version of a page to replace the cached version. in that way we can have more control over the cached version of our site. The source code of a fetched version of a page can be used to campare against the source code of the cached page and check if the webmaster have made changes. In this way the data from Webmasters Tools can be more accurate.
Sorry for my english, i speak spanish...
Thanks. I will check this feature out.
my site too has been hacked las week but i have corrected all the errors you can see at: gratuitjeux.net
Thanks for the information!!
Interesting, but it sounds reductive thinking to google fetch just as SCAM report tool.
While I was tracking down Cialis on my site, I was almost certain that Google was the one who got hacked. To anyone trying to fix their site, I would recommend using version control system, and to take a directory diff against a clean version of your site, such as an old backup, or a fresh version that you downloaded from the creator.
Anyways, now that Fetch-As-Google-Bot has helped me get my site off drugs (or rather the drugs off my site), I was wondering if perhaps GoogleBot would make rehab easy, such as a one-time quick reindexing, now that Cialis is gone.
my site too has been hacked last week but i fixed the exploits here: http://wallpaper-3d.net .
Hi everyone,
Since over a year has passed since we published this post, we're closing the comments to help us focus on the work ahead. If you still have a question or comment you'd like to discuss, free to visit and/or post your topic in our Webmaster Central Help Forum.
Thanks and take care,
The Webmaster Central Team
Post a Comment