Robots.txt file is nothing more than few lines of coding with an instructions for crawlers. Sometimes website owner don't want to index any of page or post, In such scenario the owner will have an option to restrict bots.
In Robots.txt file consists on simple commands which tells search engine bots what they have to index and what not by also assigning a maximum time in specific folders.
Recommended: How To Submit Link in Digg
In advantage of Robots.txt the top one is that you can use it also to restrict activity of crawlers of any specific search engine bots. For example in case of using any content (like yoursite.com/p/page-url.html) in blog or website and having chance of getting penalization due to violation of any strict rule of any search engine, you can restrict that specific SE crawlers to not index such content by using:
Recommended: How To Submit Link in Digg
In advantage of Robots.txt the top one is that you can use it also to restrict activity of crawlers of any specific search engine bots. For example in case of using any content (like yoursite.com/p/page-url.html) in blog or website and having chance of getting penalization due to violation of any strict rule of any search engine, you can restrict that specific SE crawlers to not index such content by using:
like for Baidu;
User-agent: baiduspider
Disallow: /p/page-url.html
It does not matter either you are on Blogger, Wordpress, or any other platform It is recommended to use Robots.txt file, as it is a good SEO practice to in order to get top rank in search results.
No comments:
Post a Comment