Following on from my last few posts I wanted to cover off the various attributes of the Webmaster Tools console from Google. In particular, I wanted to help you understand the role of a Robots.txt file and potential SEO benefits.
So what is a Robots.txt file?
A Robots.txt file is a simple text file that sits in the root of a website directory. The simple task for this file is to help control what search engine spiders/bots find and index on your site.
The specific purpose of this file however is to ‘block’ search engine spider/bots from crawling, yes that’s right ‘block’…might sound strange at first, surely it would be mad to block content from being crawled!
Turns out there are very good uses for submitting this file and it carries some SEO use, as below:
- You’re in the process of building a site or working on new pages, not wanting the incomplete work to be crawled.
- For certain websites i.e sites built using .php the Robots.txt can greatly help with SEO preventing duplicate content from being indexed. Duplicate content can be damaging to any search campaign and undermine the validity of you content.
- For large sites, Robots.txt files can help block certain pages on the site, speeding up crawl times.
- For good housekeeping it is generally recommended to use a Robots.txt file even if this is just a blank file.
It’s best to consider using this tool when setting up an online search strategy and to be aware of the potential benefits. As the amount of content we are publishing online increases over time and the way we publish this content changes, tools like this will prove very useful to help optimise it.
Within the Webmaster Tool console it is an easy step to submit a robots.txt file and I will also post some links below for sites which make it easy to create them. Note that you will need access to the root of your domain, so check with your hosting company first.
Webmaster tools – you need to have an account to access generator and checker tools.
If you have any questions about this topic, please feel free to post below.
