What is ‘robots.txt’ file?

You have private folders and membership areas on your website? Then you need this ‘robots.txt’ to stop the search engines to crawl in to your private folders. Once the search engines crawl in to your private folders, they are going to be indexed in the search engine results.

Robots.txt is a simple text file that tells the search engine robots not to crawl certain folders and pages of your web site.

When a search engine spider comes to your site first it looks for ‘robots.txt’ file in your root directory. If it is there, it reads it and follows the instructions. ALL search engines obey the robots.txt rules.

How to create it?

1. Open your Notepad or any other text editor. (Not html editor)

2. If you have a folder named ‘privatefolder’ which you don’t want to be indexed in SEs then you type like this-

User-agent:*
Disallow:/privatefolder/

‘*’ means all search engine robots. Second line tells the spiders that they are not allowed to crawl in to the ‘privatefolder’.

If you don’t want Yahoo to search ‘privatefolder’ then you can specifies it like -

User-agent: Slurp
Disallow:/privatefolder/

See the example robots.txt file here-

—————————————
# For domain: yourdomain.com

User-agent:*
Disallow:/cgi-bin/
Disallow:/privatefolder/
Disallow:/downloads/
Disallow:/important/members.html

# Edited on 29.01.09
—————————————

# indicates comments and those are for your rememberence only. Search engine spiders don’t read the lines that begin with #.
* indicates all search engines.

3. Save your text file as robots.txt on your computer. Upload it in ASCII mode in to your root directory. Then it would be like -

http://www.yourdomain.com/robots.txt

4. After uploading check the file for syntax errors at:

http://www.google.com/webmasters/tools/

Register for an account there with Google webmaster tools. Read this article:
http://www.google.com/support/webmasters/bin/answer.py?answer=35237

Google webmaster tools are very useful for optimizing your web site search engine performance.

5. TIPs:

* Never keep these lines because this bans all the search engines to spider your site and you never be listed in search engines.

User-agent:*
Disallow:/

* Always upload it in to your root directory. Should be like -
http://www.yourdomain.com/robots.txt

Author: Radhika (c)
http://www.webmasters-central.com/

Tags: disallow robots, robots txt, robots txt syntax, robots.txt, robots.txt file
Posted in: Search engine Marketing