Common mistakes in the use of SEO website optimization robots.txt _Shang pin China focuses on high quality website construction

Website production Shangpin China (ihucc.com): Myth 1. Allow covers more than Disallow and is in front of Disallow.

For example, many people wrote:

　　User-agent: *

　　Allow: /

　　Disallow: /mulu/

If you want all spiders to crawl all the pages of the site except the url below/mulu/, you are not allowed to crawl all the urls below/mulu/.

　　 SEO website optimization The search engine spider executes the rule from top to bottom, which will cause the failure of the second command. That is to say, the rule prohibiting fetching the/mulu/directory is invalid. The correct thing is that Disallow should be above Allow in order to take effect.

Myth 2. Disallow and Allow commands do not begin with a slash "/".

This is also easy to ignore. For example, someone wrote Disallow: 123.html, which has no effect. The slash "/" indicates the location of the file to the root directory. If it is not, it is impossible to determine the URL, so it is an error.

The correct one is Disallow:/123.html, so that the/123.html file can be blocked and this rule can take effect.

Myth 3. There is no slash "/" behind the masked directory.

For example, if you want to block the directory/mulu/, some people write Disallow:/mulu. Is that right?

First of all, writing (Disallow:/mulu) can indeed block all urls below/mulu/, but it also blocks all pages starting with/mulu, such as/mulu123.html. The effect is the same as Disallow:/mulu *.

The correct way is to use a slash "/" at the end if you only want to mask the directory.
This article was published by Beijing website production company Shangpin China //ihucc.com/

Common mistakes in the use of SEO website optimization robots. txt

Date：2013-04-28 Source: Shangpin China Type: website encyclopedia
Word Size： small medium big

+86 10-60259772