Posted in Uncategorized

Inside AdSense

AdSense crawler errors: Check your robots.txt file for improved ad targeting and relevancy

Thursday, November 17, 2011
This is the first post in our two-part AdSense Crawler Errors series.

There are many ways that publishers can go about optimizing their site for AdSense; opting-in to text/image ads, upgrading to our preferred ad formats, and increasing ad coverage across a site are just a few of the more well-known ones. But did you know that there’s another straightforward optimization tip that many publishers often overlook?

A bit of context
Your site’s robots.txt file essentially acts as a gatekeeper that determines which web crawlers, web robots, and search engines have access to your site and which do not. Those that are granted permission can do things like view your pages and index your site. Those that don’t have permission are not able to view or index specific sections of your site, depending on what you’ve specified.

AdSense ads are displayed through the use of anAdSense web crawler. That crawler scans your page’s content and determines which ads to display, according to specific keywords. If our AdSensecrawler is being blocked by yourrobots.txt file, we’re going to have a difficult time displaying relevant ads on your site. As a result, your users may see less relevant ads, which can lead to a lower CTR.

How you can help yourself

View the contents of yourrobots.txt file by going to[yourdomain.com]/robots.txt.  (If you have a subdomain, it likely has a robots.txt file as well, located at[sub.yourdomain.com]/robots.txt.) Be sure that the file is configured to allow our AdSense ad crawler to view your site. You can do that by simply adding the following two lines to the very top of the file:

User-agent: Mediapartners-Google
Disallow:

This will ensure that our AdSense ad crawler can access your site and will help display more relevant ads. As a result, you can potentially benefit from increased ad revenue. Please note that making this change will not impact your Google search rankings. Adding these two lines to your robots.txt file will only help to deliver better, more relevant ads to pages with AdSense code already on them. Pages that don’t have AdSense ad code will not be affected.

If you have URLs with any errors, you can see what they are by logging into your AdSense account and clicking on ‘Account Settings’ from the home page. From there, click on ‘View errors’ under ‘Access and Authorization.’

Stay tuned for the second post in ourAdSense Crawler Errorsseries, where we’ll cover crawler login issues and how you can solve them.

Advertisements

Author:

Blogger blogger blogger man I am spider of internet

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: