WordPress search engine seo website optimization crawl record code

I have been blogging for some time（ Beijing website construction ）Why do search engines not include your pages? Want to know which spiders visit your website every day? As a wordpress user（ Beijing website production ）, it is necessary to know which spiders have crawled your website every day, so as to understand the crawling frequency of spiders in each search engine and target the website SEO website optimization 。

In fact, it is very simple. Just add the following code, and then call the file code. Is it convenient? Let's get started.

I also found several spider crawling recording tools in PHP before, and the results were unsatisfactory. Moreover, most of these PHP programs need to be installed, and it is too troublesome to add spider crawl records to MYSQL. Then look for a simple spider crawl recorder~

googlebot

1. First, create a robots.php file in the root directory of the wordpress theme, and write the following contents:

<? php
function get_naps_bot()
{
$useragent = strtolower($_SERVER['HTTP_USER_AGENT']);
if (strpos($useragent, ’googlebot’) !== false){
return ’Googlebot’;
}
if (strpos($useragent, ’msnbot’) !== false){
return ’MSNbot’;
}
if (strpos($useragent, ’slurp’) !== false){
return ’Yahoobot’;
}
if (strpos($useragent, ’baiduspider’) !== false){
return ’Baiduspider’;
}
if (strpos($useragent, ’sohu-search’) !== false){
return ’Sohubot’;
}
if (strpos($useragent, ’lycos’) !== false){
return ’Lycos’;
}
if (strpos($useragent, ’robozilla’) !== false){
return ’Robozilla’;
}
return false;
}
function nowtime(){
$date=gmdate(”Y-n-j H:i:s”,time()+8*3600);
return $date;
}
$searchbot = get_naps_bot();
if ($searchbot) {
$tlc_thispage = addslashes($_SERVER['HTTP_USER_AGENT']);
$url=$_SERVER['HTTP_REFERER'];
$file=”robotslogs.txt”;
$time=nowtime();
$data=fopen($file,”a”);
fwrite($data,”Time:$time robot:$searchbot URL:$tlc_thispage\n”);
fclose($data);
}
?>

Upload it in your theme directory.

2. Add the following code at the appropriate location of Footer.php or header.php to call robots.php.

<? php include(’robots.php’) ?>

Program principle: By judging the spider identifier (such as Baidu spider and Googlebot), record the spider crawl time, and generate a log file robotslogs.txt in the root directory.

Program shortcomings: The page crawled by spiders cannot be recorded. The function is simple.