What is 'Robots.txt'?

You have private folders and membership areas on your website?Then you need this 'Robots.txt' to stop the search engines to crawl in to your private folders.Once the search engines crawl in to your private folders, they are going to be indexed in the search engine results.

Robots.txt is a simple text file that tells the search engine robots not to crawl certain folders and pages of your web site.

When a search engine spider comes to your site first it looks for 'Robots.txt' file in your root directory.If it is there, it reads it and follows the instructions.ALL search engines obey the robots.txt rules.

How to create it?

1.Open your Notepad or any other text editor.(Not html editor)

2.If you have a folder named 'privatefolder' which you don't want to be crawled by SEs then you type like this-

User-agent:*
Disallow:/privatefolder/

* means all search engine robots.Second line tells the spiders that they are not allowed to crawl in to the 'privatefolder'.

If you don't want Altavista to search 'privatefolder' then you can specifies it like-

User-agent:Altavista
Disallow:/privatefolder/

See the example robots.txt file here-

---------------------------------------
# For domain: yourdomain.com

User-agent:*
Disallow:/cgi-bin/
Disallow:/privatefolder/
Disallow:/downloads/
Disallow:/important/members.html

# Edited on 29.01.02
---------------------------------------

# indicates comments and those are for your rememberence only. Search engine spiders don't read the lines that begin with #.

* indicates all search engines.

3.Save your text file as robots.txt on your computer.Upload it in ASCII mode in to your
root directory.Then it would be like-

http://www.yourdomain.com/robots.txt

4.After uploading check the file for syntax errors at http://www.searchengineworld.com/cgi-bin/robotcheck.cgi

5.TIPs:

===Never keep these lines-

User-agent:*
Disallow:/

Because this bans all the search engines to spider your site and you never be listed in search engines.

===Always upload it in to your root directory.Should be like http://www.yourdomain.com/robots.txt.