踢掉google和baidu爬虫!
发布时间：2013-07-18 22:56:51 来源：51推一把
【摘要】在网站根目录下建立robots.txt文件,写入:User-agent: GoogleBotDisallow: *User-agent: googleBotDisallow: *User-agent: BaiduspiderDisallow: *User-agent: baiduspiderDisallow: User-agent: GoogleBotDisallow: *User-agent: googleBotDisallow: * User-
在网站根目录下建立robots.txt文件,写入:
User-agent: GoogleBotDisallow: *User-agent: googleBotDisallow: *User-agent: BaiduspiderDisallow: *User-agent: baiduspiderDisallow:
User-agent: GoogleBotDisallow: *
User-agent: googleBotDisallow: *
User-agent: BaiduspiderDisallow: *
User-agent: baiduspiderDisallow: *