Sunday, April 13, 2014

Forcing Google to NOT Crawl and Index Mobile Site

Let's assume you've got a desktop site and a mobile version on different subdomains or subdirectories. Google might actually index your mobile site's URLs and serve them up in desktop search results... yes, I have been a victim of this and seen it myself (see screenshot below). It is surprising, but Google can't get this right 100% of the time.
Desktop search result with mobile URL in Google index
So what do you do?

You have two main options to force Google desktop to crawl your desktop site and Google mobile to crawl your mobile site. Setup Webmaster Tools to specify this setup and/or modify your robots.txt. The Webmaster Tools option pretty much consists of registering both sites in Webmaster Tools and setting up distinct sitemaps. This still leaves a lot to chance assuming Google will take these directions when indexing your URLs. Also, this only works if your mobile site is on a different subdomain. If your mobile site resides in a subdirectory, like http://www.gtautomax.com's mobile site (http://www.gtautomax.com/mobile/), then you are left with robots.txt modification as your main option.

The strategy is to direct Googlebot and other desktop search bots away from the mobile site and allow Googlebot-Mobile and other mobile search bots to access the mobile site. (This assumes your site is already redirecting users from desktop URLs to mobile URLs automatically.)


If you have different desktop/mobile subdomains, your robots.txt will look something like this:

Desktop Site:
User-agent: Googlebot
User-agent: Slurp
User-agent: bingbot
Allow: /

User-agent: Googlebot-Mobile
User-Agent: YahooSeeker/M1A1-R2D2
User-Agent: MSNBOT_Mobile
Disallow: /

Mobile Site:
User-agent: Googlebot
User-agent: Slurp
User-agent: bingbot
Disallow: /

User-agent: Googlebot-Mobile
User-Agent: YahooSeeker/M1A1-R2D2
User-Agent: MSNBOT_Mobile
Allow: /


If your desktop/mobile sites are differentiated by subdirectory, then your robots.txt will look something like this:

Desktop Site:
User-agent: Googlebot
User-agent: Slurp
User-agent: bingbot
Disallow: /mobile/

Mobile Site:
User-agent: Googlebot-Mobile
User-Agent: YahooSeeker/M1A1-R2D2
User-Agent: MSNBOT_Mobile
Allow: /mobile/


As always, make sure and check your robots.txt file in Webmaster Tools for any errors and to ensure mobile URLs and desktop URLs are handled properly.