SEO experts are having problems with how not to have some of their web pages indexed by the search engine spiders. For instance, the site has two or three versions of web pages; you wish to have crawled and indexed only one of them. Or perhaps, you don’t want some web pages that contain sensitive data to appear in search engine results page. Here’s a great deal; robots.txt is the key in preventing them from getting crawled and indexed.

Robots.txt is not a firewall or a password protection tool. It is a file that simply instructs search engine spiders. It is more of a note or sign for search engines that says: “Do not crawl here.”

It is important to know where the robots.txt file should be placed. According to some of the SEO experts, robots.txt file must be located in the main directory to enable search engine spiders or user-agent find it easily. If the robots.txt is not found in the main directory, the user-agent would assume that the entire site should be crawled and indexed.

People who venture on internet marketing must know that importance and use of robots.txt to their site. Aside from keeping web pages from being shown in search results, it also helps you save bandwidth by not including other files from indexing such as JavaScript, images and others.