Some usage of robots.txt - Website Application - Discoverer Blog

The robots.txt is a plain text file. By declaring the parts of a website that you do not want robots to access in this file, you can prevent all or part of your website from being indexed by search engines, or you can specify that search engines only index certain content.

The robots.txt must be placed in the root directory of a site, and the filename must be entirely in lowercase.

Here are some basic uses of robots.txt:

l Prohibit all search engines from accessing any part of the website:

User-agent: *

Disallow: /

This article is from Discoverer Blog http://www.find321.cn/, original link: http://www.find321.cn/post/166.html