Search Engine Optimization is often termed as a very simple and easy thing but in fact in involves so many tricky things which you can’t neglect at any cost. You will have to take care of all such tiny things else you are more likely to get banned by Google. One of such tiny but very important thing is Robots.txt file.
Robots.txt file is no doubt a small text file residing in your domain root folder but you can’t neglect the importance of this file. It had no such importance when the concept of dynamic websites was not introduced. At that time SEO wasn’t also that much developed field as Google was just a child at that time. Now as the concept of static website gone out of sight, the Robots.txt became very important. In a dynamic website, there are many files, images and pages which are not shown to any user. But when Google crawls a website, it indexes everything. So if it indexed something that we didn’t want to be shown to the user, it will be visible to them. To stop crawlers and search engines from indexing some of the files, pages and images etc, we use robots.txt file. It contains instructions for crawlers which guide them what to crawl and what to leave. It may have different instructions for different crawlers and bots.
As stated above that a dynamic website has many things which should be indexed by the search engines. Such pages might have duplicate content or something which should not be shown to the users. Such pages may have some script, some broken links, some links, or something which may be against the SEO rules and policy. If we will not robots.txt file and will not stop bots from indexing such pages, Google may consider these pages a part of our content pages. When it will find some undesired scripts, broken links, duplicate content or something which is against SEO policy then it may derank your web pages or in worse case it may penalize your website too.
If you use Google or Bing Webmaster Tools then you will know that how important your robots.txt is in the eyes of search engines. Just to check, remove robots.txt from your domain root folder. Send the crawler to your website and then you will see that you will start getting error reports and most of them will be serious one. If you are a webmaster or SEO expert, you might have the idea about how critical these errors are. These can impact the ranking of your website if not corrected in time. So here you will get idea that your robots.txt file stops so many errors and hence save your ranking in Search Engines.