踢掉google和baidu爬虫!

2013年7月18日历史内容整理

在网站根目录下建立robots.txt文件,写入:User-agent: GoogleBotDisallow: *User-agent: googleBotDisallow: *User-agent: BaiduspiderDisallow: *User-agent: baiduspiderDisallow:  User-agent: GoogleBotDisallow: *User-agent: googleBotDisallow: * User-

在网站根目录下建立robots.txt文件,写入:

User-agent: GoogleBotDisallow: *User-agent: googleBotDisallow: *User-agent: BaiduspiderDisallow: *User-agent: baiduspiderDisallow:

User-agent: GoogleBotDisallow: *
User-agent: googleBotDisallow: *

User-agent: BaiduspiderDisallow: *
User-agent: baiduspiderDisallow: *