当前位置: 首页 > 常见问题 > iis屏蔽yodaobot蜘蛛,如何使用伪静态规则拦截蜘蛛访问?

iis屏蔽yodaobot蜘蛛,如何使用伪静态规则拦截蜘蛛访问?

关键词:iis屏蔽yodaobot蜘蛛   时间:2017-09-15      来源:VPS之家    点击:
Linux下 规则文件.htaccess(手工创建.htaccess文件到站点根目录)

<IfModule mod_rewrite.c>
RewriteEngine On
#Block spider
RewriteCond %{HTTP_USER_AGENT} "Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot
|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl
|Python|Wget|Xenu|ZmEu" [NC]
RewriteRule !(^robots\.txt$) - [F]
</IfModule>

windows下 规则文件httpd.conf   (本司伪静态版本为: Isapi_Rewite3.1 )

#Block spider
RewriteCond %{HTTP_USER_AGENT} "Webdup|AcoonBot|AhrefsBot|Ezooms|EdisterBot|EC2LinkFinder|jikespider|Purebot|MJ12bot
|WangIDSpider|WBSearchBot|Wotbox|xbfMozilla|Yottaa|YandexBot|Jorgee|SWEBot|spbot|TurnitinBot-Agent|mail.RU|curl|perl
|Python|Wget|Xenu|ZmEu" [NC]
RewriteRule !(^/robots\.txt$) - [F]

注:规则中默认屏蔽部分不明蜘蛛,要屏蔽其他蜘蛛按规则添加即可
附各大蜘蛛名字:
google蜘蛛: googlebot
百度蜘蛛:baiduspider
yahoo蜘蛛:slurp
alexa蜘蛛:ia_archiver
msn蜘蛛:msnbot
bing蜘蛛:bingbot
altavista蜘蛛:scooter
lycos蜘蛛: lycos_spider_(t-rex)
alltheweb蜘蛛: fast-webcrawler
inktomi蜘蛛: slurp
有道蜘蛛:YodaoBot和OutfoxBot
热土蜘蛛:Adminrtspider
搜狗蜘蛛:sogou spider
SOSO蜘蛛:sosospider
360搜蜘蛛:360spider
 

本文链接:http://www.kyvps.com/changjianwenti/1916.html"iis屏蔽yodaobot蜘蛛"转载请注明出处。

百度新闻 百度地图 谷歌地图
Copyright© 2010-2020 VPS之家 免费VPS VPS技术 IDC服务商