有人对构建服务器的虚拟机性能进行了基准测试吗？
我们一直在尝试使用虚拟机来构建服务器。我们的构建服务器都运行 WinXP32，并将它们托管在运行 Ubuntu 9.10 的 VMWare Server 2.0 上。我们构建了 C、C++、Python 包和其他各种部署任务（安装程序、7z 文件、存档等）的组合。使用 VMWare 托管构建服务器进行管理非常棒。我们可以移动它们，在一个大型 8 核机器上共享系统资源，通过 Web 界面远程访问系统，并且基本上可以更好地管理事物。
但问题是，与使用物理机器相比，性能似乎从糟糕到可怕不等，具体取决于具体日期。事实证明这非常令人沮丧。有时主机的系统负载会超过 20，有时会低于 1。这似乎与系统上实际完成的工作量无关。我怀疑系统存在瓶颈，但我似乎无法弄清楚它是什么。 （最近的怀疑是 I/O，但我们有一个专用的 1TB 7200RPM SATA 2 驱动器，带有 32MB 缓存，除了虚拟机之外什么也不做。对于 1-2 台机器来说似乎足够了。所有其他规格似乎也足够了。8GB RAM ，每个虚拟机 2GB，8 个核心，每个虚拟机 1 个）。
所以在用尽我能想到的一切之后，我想转向 Stack Overflow 社区。
- 是否有人运行或见过其他人在虚拟机内运行软件构建性能基准测试。
- 相对于物理系统，我们应该期待什么？
- 我们要放弃多少性能？
- 人们使用什么硬件/虚拟机服务器配置？
任何帮助将不胜感激。
We have been trying to use virtual machines for build servers. Our build servers are all running WinXP32 and we are hosting them on VMWare Server 2.0 running on Ubuntu 9.10. We build a mix of C, C++, python packages, and other various deployment tasks (installers, 7z files, archives, etc). The management using VMWare hosted build servers is great. We can move them around, shared system resources on one large 8-core box, remotely access the systems through a web interface, and just basically manage things better.
But the problem is that the performance compared to using a physical machine seems to range from bad to horrid depending upon what day it is. It has proven very frustrating. Sometimes the system load for the host will go above 20 and some times it will be below 1. It doesn't seem to be based on how much work is actually being done on the systems. I suspect there is a bottleneck in the system, but I can't seem to figure out what it is. (most recent suspect is I/O, but we have a dedicated 1TB 7200RPM SATA 2 drive with 32MB of cache doing nothing but the virtual machines. Seems like enough for 1-2 machines. All other specs seem to be enough too. 8GB RAM, 2GB per VM, 8 cores, 1 per vm).
So after exhausting everything I can think of, I wanted to turn to the Stack Overflow community.
- Has anyone run or seen anyone else run benchmarks of software build performance within a VM.
- What should we expect relative to a physical system?
- How much performance are we giving up?
- What hardware / vm server configurations are people using?
Any help would be greatly appreciated.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
磁盘 IO 绝对是一个问题，当您使用单个主轴进行备份时，您无法执行任何大量的磁盘 IO 活动。单个 SATA 驱动器上的 32MB 缓存将因主机和几个来宾操作系统的运行而饱和。如果您查看 Ubuntu 主机操作系统中的磁盘队列长度计数器，您应该会发现它很高（在具有 2 个驱动器的系统上，任何时间长度高于 1 的值都意味着有东西正在等待该磁盘）。
当我调整虚拟机基础设施的大小时，我通常将每个虚拟机 30-50 IOPS 的平均值作为平均值，这适用于不经常使用磁盘子系统的系统。对于不需要大量 IO 活动的系统，您可以稍微降低一些，但构建系统的 IO 模式将严重偏向于大量非常随机的相当小的读取。为了使问题更加复杂，您需要同时构建大量虚拟机，这将导致磁盘争用达到顶峰。总体磁盘带宽可能不是一个大问题（当 IO 模式完全连续时，SATA 驱动器可能可以推动 70-100Meg/秒），但是当文件很小且分散时，您的 IO 会受到主轴的限制，这将是在 7.2k SATA 上每秒大约 70-100 IO。运行 2 类虚拟机管理程序（例如具有单个来宾的 VMware Server）的主机操作系统可能会在轻负载下达到此要求。
我的建议是使用更小且速度更快的驱动器构建 RAID 10 阵列。 10k SAS 驱动器每个可为您提供 100-150 IOP，因此 4 个一组可在达到最高值之前处理 600 读取 IOPS 和 300 写入 IOPS。另请确保为驱动器托管对齐所有数据分区如果您将 VM 文件放在 RAID 阵列上，则位于 VMDK 和来宾操作系统中。对于此类工作负载，磁盘性能将提高 20-30%。对于此类情况，请避免使用 RAID 5，空间很便宜，而且 RAID 5 的写入损失意味着 RAID 5 包中需要 4 个驱动器才能达到单个驱动器的写入性能。
我要补充的另一点是，就性能而言，VMware Server 并不是一个出色的虚拟机管理程序，如果可能的话，可以迁移到类型 1 虚拟机管理程序（如 ESXi v4，它也是免费的）。设置起来并不简单，您会完全失去主机操作系统，因此这可能是一个问题，但您会看到全面更好的 IO 性能，特别是对于磁盘和网络流量。
已编辑以回复您的评论。
1) 查看您现有的 Ubuntu 主机是否确实存在问题。
我看到您已经尝试过 dstat，我认为它没有为您提供足够的详细信息来了解正在发生的情况，但我不熟悉使用它，所以我可能是错的。 iostat 可以让您很好地了解正在发生的情况 - 这篇关于使用 iostat 的文章将帮助您更好地了解击中磁盘的实际 IO 模式 - http://bhavin.directi.com/iostat-and-disk-utilization-monitoring-nirvana/ 。 avgrq-sz 和 avgwq-sz 是排队请求数量的原始指示器。高数字通常是不好的，但实际不好的情况因磁盘类型和 RAID 几何结构而异。您最终感兴趣的是查看磁盘 IO 在队列中花费的时间是否比实际接受服务的时间多\增加。计算
(await-svctim)/await*100真正告诉您您的磁盘是否正在努力跟上，是否超过 50%，以及您的 IO 的排队时间与磁盘服务的时间一样长，如果接近 100%，磁盘就会完全崩溃。如果您确实发现主机实际上并没有受到压力，并且 VMware Server 实际上只是很糟糕（很可能是这样，我从未在 Linux 平台上使用过它），那么您可能需要先尝试一种替代方案，例如 VirtualBox跳到 ESXi 上。2) 弄清楚你需要什么。
在具有良好\可接受性能的系统上确定典型构建的 IO 要求基线 - 在 Windows 上查看 IOPS 计数器 - 磁盘读取/秒和磁盘写入/秒计数器，并确保平均队列长度 <1。您需要知道系统加载时的峰值，如果所有内容都来自磁盘缓存，则瞬时峰值可能会非常高，因此请注意一分钟左右的持续峰值。一旦获得这些数字，您就可以确定一个磁盘子系统来满足您的需求。您需要查看 IO 数字的原因是，它们反映了驱动器磁头为完成读取和写入而必须经历的实际切换（每秒 IO，IOPS），除非您正在执行大型文件流或全磁盘备份它们将最准确地反映磁盘在负载下将达到的限制。
现代磁盘大约可以承受以下性能：
请注意，这些是典型驱动器的近似数字，代表驱动器在以下情况下的饱和能力：不利 IO 模式的最大负载。这是针对最坏情况的设计，除非您真的知道自己在做什么，否则您应该这样做。
RAID 包允许您并行化 IO 工作负载，并且使用合适的 RAID 控制器，N 驱动器 RAID 包将为您提供 N*（1 个磁盘的基本 IOPS）用于读取 IO。对于写入 IO，RAID 策略会造成惩罚 - RAID 0 没有惩罚，写入与读取一样快。 RAID 5 需要每个 IO 2 次读取和 2 次写入（读取奇偶校验、读取现有块、写入新奇偶校验、写入新块），因此其惩罚为 4。RAID 10 的惩罚为 2（每个 IO 2 次写入）。 RAID 6 的惩罚为 5。要计算出 RAID 阵列需要多少 IOPS，您需要获取操作系统所需的基本读取 IOPS 数，并将操作系统所需的写入 IOPS 数与相关惩罚因子的乘积相加。
3) 现在计算出满足您性能需求的 RAID 阵列结构
如果对物理基准系统的分析表明您只需要 4\5 IOPS，那么您的单个驱动器可能没问题。如果确实如此，我会感到惊讶，但不要相信我的话 - 获取您的数据并做出明智的决定。
无论如何，我们假设您在基准练习期间测量了 30 个读取 IOPS 和 20 个写入 IOPS，并且您希望能够支持这些构建系统的 8 个实例作为虚拟机。为了实现这一点，您的磁盘子系统需要能够支持操作系统 240 次读取 IOPS 和 160 次写入 IOPS。调整您自己的计算以适应您真正需要的系统数量。
如果您选择 RAID 10（我强烈鼓励它，RAID 10 会牺牲容量以换取性能，但是当您设计足够的性能时，您可以调整磁盘大小以获得所需的容量，结果通常会比 RAID5 便宜，除非您的 IO 模式涉及很少写入）您的磁盘总共需要能够提供 560 IOPS（读取 240 次，写入 320 次，以解决 RAID 10 写入惩罚因子 2）。
这需要：
- 4 个 15k SAS 驱动器
- 6 个 10k SAS 驱动器（向上取整，RAID 10 需要均匀数量的驱动器）
- 8 个 7.2k SATA 驱动器
如果您选择 RAID 5，则必须针对增加的写入损失进行调整，因此需要 880 IOPS 才能提供您想要的性能。
这需要：
- 6 个 15k SAS 驱动器
- 8 个 10k SAS 驱动器
- 14 个 7.2k SATA 驱动器
这样您将拥有更多的空间，但成本几乎是原来的两倍，因为您需要更多的驱动器，并且需要一个相当大的盒子来容纳这些驱动器。这就是为什么如果性能有任何问题的话我强烈推荐 RAID 10。
另一种选择是找到一个具有足够存储空间来满足您需求的优质 SSD（例如 Intel X-25E，而不是 X-25M 或任何更便宜的产品）。购买两个并将它们设置为 RAID 1，SSD 相当不错，但它们的故障率（即使对于像 X-25E 这样的驱动器）目前比旋转磁盘更糟糕，因此除非您准备好处理死机系统，否则您需要 RAID 1最低限度。结合良好的高端控制器（例如 X-25E），在现实世界中可以轻松维持 6k IOPS，相当于 30 个 15k SAS 驱动器。 SSD 每 GB 容量相当昂贵，但如果使用得当，它们可以为 IO 密集型任务提供更具成本效益的解决方案。
Disk IO is definitely a problem here, you just can't do any significant amount of disk IO activity when you're backing it up with a single spindle. The 32MB cache on a single SATA drive is going to be saturated just by your Host and a couple of Guest OS's ticking over. If you look at the disk queue length counter in your Ubuntu Host OS you should see that it is high (anything above 1 on this system with 2 drive for any length of time means something is waiting for that disk).
When I'm sizing infrastructure for VM's I generally take a ballpark of 30-50 IOPS per VM as an average, and that's for systems that do not exercise the disk subsystem very much. For systems that don't require a lot of IO activity you can drop down a bit but the IO patterns for build systems will be heavily biased towards lots of very random fairly small reads. To compound the issue you want a lot of those VM's building concurrently which will drive contention for the disk through the roof. Overall disk bandwidth is probably not a big concern (that SATA drive can probably push 70-100Meg/sec when the IO pattern is totally sequential) but when the files are small and scattered you are IO bound by the limits of the spindle which will be about 70-100 IO per second on a 7.2k SATA. A host OS running a Type 2 Hypervisor like VMware Server with a single guest will probably hit that under a light load.
My recommendation would be to build a RAID 10 array with smaller and ideally faster drives. 10k SAS drives will give you 100-150 IOPs each so a pack of 4 can handle 600 read IOPS and 300 write IOPs before topping out. Also make sure you align all of the data partitions for the drive hosting the VMDK's and within the Guest OS's if you are putting the VM files on a RAID array. For workloads like these that will give you a 20-30% disk performance improvement. Avoid RAID 5 for something like this, space is cheap and the write penalty on RAID 5 means you need 4 drives in a RAID 5 pack to equal the write performance of a single drive.
One other point I'd add is that VMware Server is not a great Hypervisor in terms of performance, if at all possible move to a Type 1 Hypervisor (like ESXi v4, it's also free). It's not trivial to set up and you lose the Host OS completely so that might be an issue but you'll see far better IO performance across the board particularly for disk and network traffic.
Edited to respond to your comment.
1) To see whether you actually have a problem on your existing Ubuntu host.
I see you've tried dstat, I don't think it gives you enough detail to understand what's happening but I'm not familiar with using it so I might be wrong. Iostat will give you a good picture of what is going on - this article on using iostat will help you get a better picture of the actual IO pattern hitting the disk - http://bhavin.directi.com/iostat-and-disk-utilization-monitoring-nirvana/ . The avgrq-sz and avgwq-sz are the raw indicators of how many requests are queued. High numbers are generally bad but what is actually bad varies with the disk type and RAID geometry. What you are ultimately interested in is seeing whether your disk IO's are spending more\increasing time in the queue than in actually being serviced. The calculation
(await-svctim)/await*100really tells you whether your disk is struggling to keep up, above 50% and your IO's are spending as long queued as being serviced by the disk(s), if it approaches 100% the disk is getting totally slammed. If you do find that the host is not actually stressed and VMware Server is actually just lousy (which it could well be, I've never used it on a Linux platform) then you might want to try one of the alternatives like VirtualBox before you jump onto ESXi.2) To figure out what you need.
Baseline the IO requirements of a typical build on a system that has good\acceptable performance - on Windows look at the IOPS counters - Disk Reads/sec and Disk Writes/sec counters and make sure the average queue length is <1. You need to know the peak values for both while the system is loaded, instantaneous peaks could be very high if everything is coming from disk cache so watch for sustained peak values over the course of a minute or so. Once you have those numbers you can scope out a disk subsystem that will deliver what you need. The reason you need to look at the IO numbers is that they reflect the actual switching that the drive heads have to go through to complete your reads and writes (the IO's per second, IOPS) and unless you are doing large file streaming or full disk backups they will most accurately reflect the limits your disk will hit when under load.
Modern disks can sustain approximately the following:
Note these are approximate numbers for typical drives and represent the saturated capability of the drives under maximum load with unfavourable IO patterns. This is designing for worst case, which is what you should do unless you really know what you are doing.
RAID packs allow you to parallelize your IO workload and with a decent RAID controller an N drive RAID pack will give you N*(Base IOPS for 1 disk) for read IO. For write IO there is a penalty caused by the RAID policy - RAID 0 has no penalty, writes are as fast as reads. RAID 5 requires 2 reads and 2 writes per IO (read parity, read existing block, write new parity, write new block) so it has a penalty of 4. RAID 10 has a penalty of 2 (2 writes per IO). RAID 6 has a penalty of 5. To figure out how many IOPS you need from a RAID array you take the basic read IOPS number your OS needs and add to that the product of the write IOPS number the OS needs and the relevant penalty factor.
3) Now work out the structure of the RAID array that will meet your performance needs
If your analysis of a physical baseline system tells you that you only need 4\5 IOPS then your single drive might be OK. I'd be amazed if it does but don't take my word for it - get your data and make an informed decision.
Anyway let's assume you measured 30 read IOPS and 20 write IOPS during your baseline exercise and you want to be able to support 8 instances of these build systems as VM's. To deliver this your disk subsystem will need to be able to support 240 read IOPS and 160 write IOPS to the OS. Adjust your own calculations to suit the number of systems you really need.
If you choose RAID 10 (and I strongly encourage it, RAID 10 sacrifices capacity for performance but when you design for enough performance you can size the disks to get the capacity you need and the result will usually be cheaper than RAID5 unless your IO pattern involves very few writes) Your disks need to be able to deliver 560 IOPS in total (240 for read, and 320 for write in order to account for the RAID 10 write penalty factor of 2).
This would require:
- 4 15k SAS drives
- 6 10k SAS drives (round up, RAID 10 requires an even no of drives)
- 8 7.2k SATA drives
If you were to choose RAID 5 you would have to adjust for the increased write penalty and will therefore need 880 IOPS to deliver the performance you want.
That would require:
- 6 15k SAS drives
- 8 10k SAS drives
- 14 7.2k SATA drives
You'll have a lot more space this way but it will cost almost twice as much because you need so many more drives and you'll need a fairly big box to fit those into. This is why I strongly recommend RAID 10 if performance is any concern at all.
Another option is to find a good SSD (like the Intel X-25E, not the X-25M or anything cheaper) that has enough storage to meet your needs. Buy two and set them up for RAID 1, SSD's are pretty good but their failure rates (even for drives like the X-25E's) are currently worse than rotating disks so unless you are prepared to deal with a dead system you want RAID 1 at a minimum. Combined with a good high end controller something like the X-25E will easily sustain 6k IOPS in the real world, that's the equivalent of 30 15k SAS drives. SSD's are quite expensive per GB of capacity but if they are used appropriately they can deliver much more cost effective solutions for tasks that are IO intensive.