yahoo的后台用的什么集群

Question

传了３００Ｍ照片速度非常快，后台的ＩＯ性能一定非常好，不知道用的何种文件系统

Answer 1

应该是比较老的资料了~

Answer 2

98年 linux 还是不能和bsd 比的

Answer 3

什么年代的东西?

Answer 4

原帖由 HonestQiao 于 2006-7-15 16:44 发表
http://www.ictp.trieste.it/~cfon ... oo_and_FreeBSD.html

Yahoo! and FreeBSD

by David Filo, Co-Founder of Yahoo!

-------------------------- ...

http://bbs.chinaunix.net/viewthr ... mp;highlight=Yahoo!
当我们在斯坦福大学完成 Yahoo 开发的时候，使用的两台计算机分别是：运行 OSF 的 DEC Alpha 和运行 SunOS 的 Sparc 20。最初这两台机器完全能满足工作需要，但不久我们就发现所用的操作系统在设计时都不具备处理大量 HTTP 请求的能力。事实上，当时还没有任何商业系统可以解决我们所面临的扩展问题，这也是身处网络技术最前沿所必须面对的无奈现实。

　　离开斯坦福之后，我们又用过几种平台包括 SGI IRIX，Linux 和 BSDI，但其性能和稳定性都不能令人满意，我们只得继续寻找。随着 Yahoo 的日渐流行，平台的可扩展能力和稳定性是最关键的。那时我们 FreeBSD 还一无所知，但当读到有关介绍时，我决定试它一试。

　　以前我经历过许多次 PC 操作系统那恼人的漫长的安装过程，所以现在有些怀疑是否还想再用三天来安装另一个系统。然而 FreeBSD 的安装却出乎意料的顺利：到 FreeBSD 站点下载一张启动软盘映象文件，再用它引导计算机，并回答几个问题，几分钟就通过网络安装完成了。这是我所遇到的安装过程最简单省力的操作系统。

　　几天之后我安装一台 FreeBSD 服务器加进我们的 Web 服务器集群中，结果它的性能明显优于其它机器，而且运行也更稳定。试验时间持续了几个星期，我们对 FreeBSD 很满意。除了诱人的价格之外，主要应归功于 FreeBSD 的稳定性、运行性能和允许访问源代码的特点。从此，我们的产品和开发环境就几乎离不开它了。

　　初时我们还对 FreeBSD 的技术支持和发展方向心存疑问，但事实证明我们能够从其核心小组（core team）及其他用户处得到大力的支持。有了这些支持和源代码，我们遇到的问题几乎都可以立即得到解决。在过去的两年中，我们对 FreeBSD 及其工作人员有了更多的了解，也明确了它的组织结构和发展方向。

　　我们最初使用 Pentium 100 来运行 FreeBSD 2.0.5.，随后逐渐把其余的服务器也转向 FreeBSD，到现在我们有 50 多台服务器全都运行不同版本的 2.1 STABLE。同时我们还在测试 2.2版，希望能在半年内取代 2.1。我们的机型从 64MB 内存的 Pentium 100 到 256MB 内存的 PPro200 都有，当需要进行额外的 I/O 操作时，就利用能扫描多重磁盘的 ccd，网络则采用100Mbps 的以太网。整体的系统配置性价比极高。（编者注：ccd 可将多个物理磁盘镜象成一个大的逻辑磁盘）。

　　FreeBSD 运行非常稳定，我们已连续 180 天每天接受 400 万以上 HTTP 请求。在一台 128MB 内存的 PPro200 上，结合 ccd 磁盘扫描技术，我们可一天处理 1200 万 HTTP 请求。美中不足的就是，为 FreeBSD 写的第三方软件极少。尽管这种情况已在改变，但短期内也难见成效。目前唯一的途径就是由 Yahoo! 和其他公司联合起来，让软件开发商们把注意力转向这个庞大的市场。

　　为了能在快速发展的同时升级我们的服务，我们期望能利用 SMP 来达到更优的性能价格比。从这个角度看，在其它平台（如 Alpha）上运行 FreeBSD 也是可行的。FreeBSD 还可能提供更多的服务如大型的可靠的 RAID 文件服务器。总之，我们认为 FreeBSD 在性能，稳定性，技术支持方面都十分出色，在使用它两年之后，我们还未发现任何其它产品能取而代之。（编者注：98 年 10 月份正式发行的 FreeBSD 3.0 已开始支持 SMP 和 Alpha 平台）。

Answer 5

http://www.ictp.trieste.it/~cfon ... oo_and_FreeBSD.html

Yahoo! and FreeBSD

by David Filo, Co-Founder of Yahoo!

--------------------------------------------------------------------------------

Yahoo! began life at Stanford University on a DEC Alpha box running OSF and a Sparc 20 running SunOS. They served us well for the first year but we learned that neither system was really designed for handling a large number of HTTP requests. In fact we were unable to find any commercial system that addressed the problems we were facing with scalability. This was one of the unfortunate realities of being at the forefront of Web technology.

After leaving Stanford we used a few platforms including SGI IRIX, Linux, and BSDI. Not being impressed with anything we'd used (in terms of performance and stability), we were still looking for alternatives. As Yahoo! grew more popular, both scalability and stability were becoming critical to our success. At the time none of us knew anything about FreeBSD, but after seeing references to it I thought I'd give it a try.

Having spend many frustrating hours trying to install other PC OS's, I was a bit skeptical. I had no intention of spending three days trying to install yet another one. To my surprise I went to the FreeBSD Web site, downloaded the floppy boot image, booted a PC with the created floppy, answered a few install questions, and a few minutes later FreeBSD was installing over the Net. The real surprise was when I cam back later to a fully configured system that actually worked. If anything had gone wrong with that install it would likely been the end of that trial. Luckily for us that it was the easiest and most painless OS installs I had ever experienced.

A couple of days later we added a FreeBSD box to our cluster of Web servers. Not only did it out-perform the rest of our machines, but it was more stable. A few weeks into this experiment and we were sold. Although the
price was certainly attractive, it was the stability, performance, and access to the sourcode that sold us. Ever since then we've used FreeBSD almost exclusively for production as well as our development environment.

Early on the two big unknowns were support issues and the future direction of FreeBSD. The support we've received from the core team as well as other users has been excelent. This support along with the source code has allowed us to solve any issues we've had almost immediately. Likewise we were pleasantly surprised with the organization and direction of the FreeBSD project as we learned more about it and the people involved over the last two years.

We started with a single Pentium 100 box running FreeBSD 2.0.5. We eventually migrated the rest of our production servers to FreeBSD and today we have over 50 servers running various versions of 2.1 STABLE. We are in the process of testing 2.2 STABLE and hope to convert during the next 6 months. The machines we use range from a Pentium 100 with 64MB of memory to a PPro200 with 256MB of memory. When additional I/O performance is needed we use ccd with stripping over multiple disks. 100Mbps fast ethernet is used for networking. Overall an extremely cost effective solution.

FreeBSD has been extremely stable for us. We've seen over 180 days of uptime on a machine serving over 4 million HTTP requests per day. Performance has been impressive too. With disk striping using ccd we've been able to serve over 12 million HTTP requests per day on a PPro200 with 128MB of memory. One of the only negative things we've found with FreeBSD has been
the lack of third party software. Fortunately this is changing, but there's still a long way to go. The only way for this to change is for Yahoo! along with other organizations to convince the software vendors that there is a big enough market for their products.

One of our big technical challenges is scaling our services in the face of rapid growth. Looking forward we are very interested in using SMP to achieve even better price/performance. FreeBSD on other platforms (e.g. Alpha) is also interesting from the price/performance perspective. We are also looking at FreeBSD to provide other services such as large reliable RAID file servers. Overall we've found FreeBSD to excel in performance, stability, technical support, and of course price. Two years after discovering FreeBSD, we have yet to find a reason why we should switch to anything else.

David Filo,

Co-founder of Yahoo!

Taken from FreeBSD News

Issue 1

LinkExchange Member
Free Home Pages at GeoCities

Get your own Free Home Page

Answer 6

谢谢老大，感觉ＨＰＣ的前途很大，以后请多多指教．

Answer 7

没有什么不透明的.  搜搜就知道了，都是公开的.
楼上同学不要把自己消息闭塞当作别人信息的不公开。

Yahoo 的技术是由Yahoo 一个叫作 Yahoo Research的部门来完成调研和决策的.
Yahoo第一期买了HP 60T SFS GigE 方案，今年正在购买另外的100-200TB并且在年底要完成采购和安装.
HP的SFS是lustre的HP版本，HP大规模存储研发部门对lustre的代码进行了增强，针对HP的硬件和设计理念有了大幅度的提高。
60TB的第一期总共有30个OSS, MDS/ADMIN节点，每个OSS用一个 SFS20的磁盘柜.网络这边用的是Force10 network 交换机，混合了HP和RLX的刀片服务器。

负责这个项目的是HP的solution architect Keith Bazarnick, Lustre(CFS)公司出了一个人是 Jeff Denworth, 一共有11个人来自不同的公司参与了这个项目.

具体的实现细节我也有，不过就不能贴在这里了，不过上面这些信息在网上花费点时间都找得到.

Answer 8

恩，后台做的应该是相当不错的，不知道有没有可能是Lustre
个人估计ＬＩＮＵＸ集群在今后会有很大的发展

Answer 9

yahoo的技术是不透明的，但可以想到的是bsd,采用了sgi的文件系统。。
呵呵，纯属猜测，欢迎拍砖。