As we all know, before the launch of "Baidu Youah", Taobao directly blocked Baidu spiders in order to avoid malicious grabbing by Baidu, that is, the boisterous robots.txt agreement at that time.

Even if you open Taobao's robots.txt protocol, you can still see that Taobao is still blocking Baidu's spiders.

But is it true

See the figure below

It turns out that Taobao's robots.txt file has no effect on Baidu Spider.

My brother speculated that Baidu did this because it considered the strategic direction, because it found that some well-known stores ranked very well in Baidu. It is enough to prove the importance of Baidu to Taobao. What is more dramatic is that Tmall (Taobao Mall) also uses the robots.txt protocol. It also shields Baidu Spider.

Baidu also adopts the attitude of crazy grabbing, as shown in the figure

Another possibility is that the weight of Taobao and Tmall.com is too high, which leads to Baidu Spider's crawling without following the search engine agreement. However, it can be recalled that a few days ago, Yitao maliciously grabbed the commodity data of Jingdong Mall and Newegg, which also did not follow the rules of the search engine.

Until now, Jingdong Mall and others have also blocked the spiders of Yitaowang,

What I want to say is that the current search engine does not follow the rules of Internet protocol for its own benefit, which is worth reflecting on. I call on such a big Internet company to set up a banner in the entire IT industry.