作者:车东 产品经理
上周末豆瓣的阿北给我电话:最近你们雅虎的蜘蛛(Yahoo! Slurp)对豆瓣网的抓取频度非常高,导致服务器的速度有些慢,如何才能让Slurp降低抓取频度呢?
我首先想到的建议是在网站的robots.txt中增加Crawl-delay: 设置,这个设置是目前Slurp独有的,用来告诉Slurp蜘蛛2次访问之间的间隔,单位是秒。
剩下的问题就是Crawl-delay的值该设置多大,这个要看网站自身可以承受的负载,假设你希望Slurp每10秒来访问一次,这个设置就是:
User-agent: *
Crawl-delay: 10
我推荐使用 User-agent: * 万一有其他引擎逐步也支持这个配置呢,而不支持这个配置的引擎也会跳过这句。
可实际上我从自己网站的日志中看到:Slurp的压力却不止10秒一次,原因是这样:雅虎美国和雅虎中国有2套蜘蛛在运行,因此实际的访问可能是这样:
1.1.1.1 30/Jun/2006:00:00:03 Yahoo! Slurp China
2.2.2.2 30/Jun/2006:00:00:04 Yahoo! Slurp
1.1.1.1 30/Jun/2006:00:00:13 Yahoo! Slurp China
每个蜘蛛都是遵循Crawl-delay间隔的,因此为了进一步降低Slurp压力,希望达到10秒一次请求的频道 实际的Crawl-delay配置就要*2。
但是实际上我看到的雅虎的蜘蛛是分布在很多台服务器上的:而多个服务器集群之间也是不相互协调同一网站的抓取频度的
1.1.1.1 30/Jun/2006:00:00:03 Yahoo! Slurp China
2.2.2.2 30/Jun/2006:00:00:04 Yahoo! Slurp
1.1.1.1 30/Jun/2006:00:00:13 Yahoo! Slurp China
1.1.1.2 30/Jun/2006:00:00:13 Yahoo! Slurp China
1.1.1.1 30/Jun/2006:00:00:23 Yahoo! Slurp China
1.1.1.2 30/Jun/2006:00:00:27 Yahoo! Slurp China
我统计了一下:Slurp来源于40多个IP网段,同网段内的蜘蛛协调抓取频度。因此:实际的Slurp设置,需要设置为10*40 = 400秒,才能达到期望的Slurp每10秒访问一次频度。
给豆瓣推荐的Crawl-delay: 100 平均2.5秒访问一次,应该没有问题了。
另外,雅虎搜索的帮助中心刚刚改版,专门增加了站长专区,可以去看看。
注:我查了一下Slurp的意思,就是咂吧嘴的声音,发音听着很像在吃面条吧?
"I was looking at some of your articles on this website and I conceive this web site is rattling instructive! Keep on putting up."
发布者:Jonathan Leemow - 2012年05月16日 5:37I just added this weblog to my rss reader, good stuff. Can't get enough!
发布者:Liane Proffitt - 2012年04月23日 21:35I absolutely love your blog and find a lot of your post's to be precisely what I'm looking for. Do you offer guest writers to write content in your case? I wouldn't mind producing a post or elaborating on a few of the subjects you write with regards to here. Again, awesome website!
发布者:diablo 3 beta download - 2012年04月21日 1:11I simply want to say I am just beginner to weblog and seriously enjoyed your website. Likely I’m likely to bookmark your blog . You absolutely have superb articles and reviews. Thanks a lot for sharing your website.
发布者:Micheal Rubinoff - 2012年04月14日 12:48I just want to tell you that I am just very new to blogs and certainly loved you're blog. Likely I’m want to bookmark your blog post . You certainly come with exceptional writings. Thanks for sharing with us your web page.
发布者:Van Ubl - 2012年03月24日 8:04I just want to tell you that I'm very new to weblog and certainly liked your web-site. Most likely I’m going to bookmark your site . You definitely have good article content. Cheers for revealing your blog.
发布者:Gabriel Souliere - 2012年03月23日 14:42hhhhhhhhhhhhhhhhhhhhhhhhhh
发布者:vcsfdsds - 2012年01月28日 14:10Awesome writing style!
发布者:Daniel Cowee - 2012年01月22日 23:01I stumbled on your page from a second article not to mention am really focused on this discovering this, I'm strongly regarding information as well as love getting to know more using this. If practical, as you get expertise, It is most helpful for my situation. would one mind updating your site with additional information?
发布者:best registry cleaner - 2012年01月07日 13:03Your brief article 雅虎æœç´¢æ—¥å¿—: 如何控制Yahoo! Slurp蜘蛛的抓å–频度 appears to be great. I consider this really is among the most valuable articles to me. I was examining this blog site and I am amazed. I can't wait around to study a lot more coming from you due to the fact this is really a tremendous website. I’ll book mark your blog and also verify again here frequently.
发布者:ex7r4 b4links pow3r - 2011年12月10日 0:42Greetings from Ohio! I'm bored to death at work so I decided to check out your blog on my iphone during lunch break. I enjoy the info you present here and can't wait to take a look when I get home. I'm surprised at how quick your blog loaded on my phone .. I'm not even using WIFI, just 3G .. Anyways, very good site!
发布者:meg43tr44gi joe - 2011年12月10日 0:18plumbing can really be a hardwork specially if you are not very skillfull in doing home plumbing.
发布者:Caramoan Tour Package - 2011年07月01日 4:01my hard drive is getting full already because i always download free online movies on the internet;
发布者:Caramoan Tour - 2011年06月30日 12:00well, outsourcing can actually save any company several million dollars because of cheap labor'
发布者:Caramoan Island - 2011年06月30日 6:25oh well, online promotion also takes a lot of work just like offline promotion of products and services..
发布者:caramoan hotels - 2011年06月29日 6:19oh well, American Dad is a nice tv series. my sixteen year old daughter just loves watching it*
发布者:Caramoan Tour - 2011年06月29日 3:04Strange this post is totaly unrelated to what I was searching google for, but it was listed on the first page. I guess your doing something right if Google likes you enough to put you on the first page of a non related search. :)
发布者:gartenlaternen - 2011年06月13日 5:34Yes great..thank you. information I was very pleased to find this web-site.I wanted to thanks for your time for this wonderful read!! I definitely enjoying every little bit of it and I have you bookmarked to check out new stuff you blog post.
发布者:gartenlaternen - 2011年06月12日 14:40I’m really enjoying the design and layout of your website. It’s a very easy on the eyes which makes it much more enjoyable for me to come here and visit more often. Did you hire out a designer to create your theme? Outstanding work!
发布者:gartenlaternen - 2011年06月12日 3:37It is rare for me to discover something on the net that’s as entertaining and intriguing as what you have got here. Your page is sweet, your graphics are great, and what’s more, you use source that are relevant to what you are talking about. You’re definitely one in a million, keep up the good work!
发布者:belt manufacturers - 2011年06月11日 6:41I have been looking for this info for quite a while. Nearly 9 hours of continuous surfing, at last I found that in your article. I don’t understand why Bing don’t display this kind of useful websites in the first few lines. Normally the top web sites are rubbish. Perhaps it is time to change to other search engine.
发布者:manhole cover manufacturers - 2011年06月10日 12:28With havin so much content and articles do you ever run into any problems of plagorism or copyright violation? My blog has a lot of exclusive content I’ve either written myself or outsourced but it appears a lot of it is popping it up all over the internet without my agreement. Do you know any methods to help stop content from being stolen? I’d genuinely appreciate it.
发布者:belt manufacturers - 2011年06月10日 2:54I really wanted to write down a small note so as to appreciate you for these remarkable tips you are giving at this site. My time intensive internet investigation has now been compensated with incredibly good content to write about with my visitors. I would point out that many of us readers actually are very much lucky to be in a good site with so many perfect professionals with very helpful points. I feel very much fortunate to have used your site and look forward to really more entertaining minutes reading here. Thanks once more for a lot of things.
发布者:belt manufacturers - 2011年06月07日 9:46I’m really enjoying the design and layout of your website. It’s a very easy on the eyes which makes it much more enjoyable for me to come here and visit more often. Did you hire out a designer to create your theme? Outstanding work!
发布者:basisschool doetinchem - 2011年04月29日 20:19Good post. I learn one thing tougher on completely different blogs everyday. It’ll all the time be stimulating to read content from different writers and follow a bit of something from their store. I’d choose to use some with the content on my weblog whether or not you don’t mind. Natually I’ll give you a hyperlink on your web blog. Thanks for sharing.
发布者:lighting manufacturers - 2011年04月29日 0:16Thank you so much pertaining to giving everyone an update on this subject matter on your blog. Please understand that if a completely new post appears or when any improvements occur to the current publication, I would be thinking about reading more and finding out how to make good using of those strategies you share. Thanks for your efforts and consideration of other folks by making this website available.
发布者:outdoor lighting manufacturers - 2011年04月28日 23:35Thanx for the effort, keep up the good work Great work, I am going to start a small Blog Engine course work using your site I hope you enjoy blogging with the popular BlogEngine.net.Thethoughts you express are really awesome. Hope you will right some more posts.
发布者:hang hieu - 2011年04月21日 2:56This can be the exact material I ve been attempting to find. My organization is very engraved with all the moderate of data present in here. I am to endorse this kind of to anyone including all my buddies. That is well said and marvelous. Knowledge for this wonderful study. I am going to view the place again presently.
发布者:shop thoi trang - 2011年04月21日 1:51Thanx for the effort, keep up the good work Great work, I am going to start a small Blog Engine course work using your site I hope you enjoy blogging with the popular BlogEngine.net.Thethoughts you express are really awesome. Hope you will right some more posts.
发布者:thoi trang 2011 - 2011年04月21日 0:17Just got done looking for this topic in the major article directories and was extremely disappointed by the lack of relevant information. Then a friend pointed out your blog to me and I must say I am absolutely blown away by the level of understanding you demonstrated on the subject.
发布者:pyjama kinderen - 2011年04月18日 12:18Hi, I think your website might be having browser compatibility issues. When I look at your blog in Opera, it looks fine but when opening in Internet Explorer, it has some overlapping. I just wanted to give you a quick heads up! Other then that, fantastic blog!
发布者:pyjama - 2011年04月18日 6:28I'm curious to find out what blog system you're working with? I'm experiencing some small security problems with my latest blog and I'd like to find something more safeguarded. Do you have any recommendations?
发布者:pyjama - 2011年04月18日 5:31Hey I definetly like this story and it was so marvelous and I am surely going to save it. One thing to say the exceptional analysis you have done is definetly extraordinary !! Who does that additional research these days? Bravo :) Just one more tip you shouldset up any Translator for your Global Users :)
发布者:pyjama - 2011年04月18日 4:54I guess you have made a few really fascinating points. Not as well many people would really think about this the direction you just did. I am truly impressed that there is so much about this subject that has been unveiled and you made it so nicely, with so considerably class. Superior one, man! Really special things right here.
发布者:tuinlantaarn - 2011年04月17日 20:06Hi, I think your website might be having browser compatibility issues. When I look at your blog in Opera, it looks fine but when opening in Internet Explorer, it has some overlapping. I just wanted to give you a quick heads up! Other then that, fantastic blog!
发布者:buitenlantaarn - 2011年04月17日 15:57I actually wanted to send a small remark so as to express gratitude to you for the wonderful pointers you are giving on this website. My extended internet investigation has at the end of the day been compensated with sensible tips to write about with my colleagues. I would suppose that many of us visitors are very much endowed to dwell in a fine site with very many marvellous people with very beneficial plans. I feel really happy to have discovered the site and look forward to plenty of more fabulous times reading here. Thanks once more for all the details.
发布者:tuinlantaarn - 2011年04月16日 17:47It is rare for me to discover something on the net that’s as entertaining and intriguing as what you have got here. Your page is sweet, your graphics are great, and what’s more, you use source that are relevant to what you are talking about. You’re definitely one in a million, keep up the good work!
发布者:kinderpyjama - 2011年04月16日 12:10Took me time to read all the responses, but I truly loved the article. It turned out to be very useful to me and I am sure to all the commenters here! It’s continually awesome when you can not only be informed, but also entertained! I’m sure you had fun writing this article. Regards, Clotilde.
发布者:tuinlantaarn - 2011年04月14日 14:30I am just commenting to make you understand of the useful discovery my wife’s child enjoyed viewing your web site. She came to understand lots of details, which include what it’s like to possess an ideal teaching character to let other people very easily learn a number of tortuous things. You undoubtedly surpassed her expected results. I appreciate you for presenting such informative, safe, explanatory and even easy guidance on this topic to Mary.
发布者:zijden pyjama - 2011年04月12日 9:34Hi could I use some of the information here in this post if I provide a link back to your site? Richard =)
发布者:vietnam manufacturers - 2011年03月30日 9:30Been searching for content like this for my questionnaire I'm working on. Thank you very much =)
发布者:herren nachthemd - 2011年03月29日 20:57Hi could I copy a few lines here in this post if I provide a link back to your site? Thnx =)
发布者:pyjama - 2011年03月29日 14:20I rarely reply to posts but I will in this case, to give some credit. I will bookmark this :-)
发布者:thoi trang 2011 - 2011年03月28日 9:13I was searching for something else but this captured my attention =)
发布者:klassieke buitenlampen - 2011年03月26日 8:26My friend and I have been looking for content like this for my little research project I'm working on. Thank you very much =)
发布者:pyjama - 2011年03月24日 5:30I normally do not comment to posts but I will in this case, to show some credit. I will bookmark this :)
发布者:lantaarnpaal - 2011年03月23日 15:24我们的网站怎么设置这个标准呢,你去看看吧谢谢,是一个台服魔兽加速器http://lonlife.net
发布者:丢了 - 2010年11月10日 15:57碰到同样问题,谢谢说明 <a href="http://www.0931ps.com">http://www.0931ps.com</a>
发布者:Genio - 2010年09月18日 21:47呵呵。。学到东东....
还有一个问题就是网络日志url怎么看,谢谢?
呵呵,我今天到空间日志看了,雅虎有来哦,谢谢光临。http://www.8821.info/
发布者:池州SEO - 2010年05月28日 15:31碰到同样问题, http://www.2lengzi.cn
发布者:疯雨 - 2010年05月10日 14:26要是都能支持就好了
http://www.7c2c.com
www.bnnet.cn
发布者:feng - 2010年04月21日 14:12顶。写的太好了
发布者:google排名 - 2009年12月13日 11:35没有搜索框的搜索
http://www.ta6678.cn
http://www.xxlsc.cn
已经转载
发布者:seog - 2009年05月02日 22:59http://seo-dic.com.cn/archives/1172
晕,不能贴地址么
<a href="http://seo-dic.com.cn/archives/1172">搜索想说爱你很容易 如何控制搜索引擎的行为 </a>,美丽星点收集的,有你们博客里的内容也,嘻嘻
发布者:美丽星点 - 2008年10月15日 16:04嗯,有学习到东西,robots这个东西设置以后确实对我的站的服务器负荷减少了
发布者:在洋之舟 - 2008年08月07日 9:02现在最不会的就是SEO,一点都不会,谁会呀,不优化可怎么嫩提高收录了
发布者:小菜 - 2008年05月03日 6:01我是垃圾我怕谁?
发布者:我是垃圾我怕谁? - 2008年04月01日 10:38为什么我网站SO不到多少内容呢,还有robots.txt这个文件是要传到服务器上的吗
发布者:阿Q在线www.rqline.cn - 2008年03月16日 17:49快受不了,yahoo!slurp已经严重干扰偶论坛的运行,每周内必有几天游客暴增,检查ip均为雅虎中国,偶已经从后台封掉了雅虎,现在准备用robots.txt来完全禁止掉yahoo。
发布者:某bbs管理员 - 2007年04月30日 1:18請問可不可以改成 META 標籤方式呈現
发布者:Eden - 2007年02月10日 17:48你好.我想问一下,我的网站是半个月前提交的,每天都看见IIS日志里有雅虎的蜘蛛爬网页,但一直到现在yahoo.cn用site:www.xkxz.com也查不到我的站点信息,而选中全部语言可以看到首页被收录,其他很多信息都不是我本站的信息,这是怎么回事??
发布者:凌云 - 2006年12月19日 12:03不错的功能
发布者:最新更新 - 2006年12月04日 16:22雅虎中国服务器群来的频率太高了,高峰时一秒内来三次,真象是攻击行为,不知道有没有更好的法,如指定时间段来访。nc2000.net
发布者:nc2000.net - 2006年08月05日 9:31那雅虎针对不同的网站的更新频率也不一样吧!不一样的依据又是什么呢?还请指教哦!
发布者:慕以 - 2006年07月07日 9:33恩 又学到了一些.
看来关键是要统计一下分布式crawler的个数来加倍delay了
不过对于不支持这项设置的crawler还是没办法
改了设置不是马上生效吧?
应该也有个刷新时间值,应该是下次采集开始才更新频度值.
但如果40多个爬虫恰好都在同一个时间段内采集,即便每只间隔100秒,40只先后脚的来爬,到最后一只爬完后,第一只的间隔时间又到了,造成的压力就有DDOS的效果了,不过这可能是最"理想"状态下才会发生的.
发布者:RainChen - 2006年07月06日 12:08终于见到传说中的雅虎大牛车东露面了,沙发一个!
发布者:慕泥黑 - 2006年07月06日 11:26修改完了可以验证一下 robots.txt 是否有效
发布者:Fenng - 2006年07月06日 10:59