使用 Linux 虚拟服务器实现 MMO 游戏中区域的负载平衡
我是一名 MMO 游戏开发人员，目前我们公司面临一些可扩展性问题，我认为这些问题可以通过游戏世界的适当集群来解决。
我真的不想重新发明轮子，这就是为什么我认为 Linux 虚拟服务器可能是一个不错的选择，尤其是使用一些 7 级负载平衡技术。
我目前正在将 ktcpvs 作为负载平衡解决方案，想知道它是否是一个正确的选择。
主要思想是在专用服务器上运行多个区域（我的游戏中的“位置”）。 当玩家决定前往某个特定位置时，负载均衡器会决定哪个区域服务器将实际为玩家提供服务（这实际上就是我需要 7 级负载均衡器的原因），
大家对上述内容有何看法？
更新：我向 LVS 用户邮件列表发布了同样的问题 http://marc.info/?l=linux-virtual-server&m=124976265209769&w=2
更新： 我也开始了类似的主题gamedev.net 论坛 http://www.gamedev.net/community /forums/topic.asp?topic_id=544386
I'm a developer of a MMO game and currently we're at my company facing some scalability issues which, I think, can be resolved with proper clustering of the game world.
I don't really want to reinvent the wheel that's why I think Linux Virtual Server could be a good choice especially with some Level 7 load balancing technique.
I'm currently looking at ktcpvs as a load balancing solution and wonder if it's a proper choice.
The main idea is to have a number of zones("locations" in terms of my game) running on dedicated servers. When a player decides to go to some specific location the load balancer decides which zone server will be actually serving the player(that's actually why I need a Level 7 load balancer)
What do you folks think about all said above?
Update: I posted the same question to LVS users mailing list http://marc.info/?l=linux-virtual-server&m=124976265209769&w=2
Update: I also started the similar topic on the gamedev.net forum http://www.gamedev.net/community/forums/topic.asp?topic_id=544386
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（4）
为了解决您的问题，我们需要了解您是否需要音量或响应，但很难同时获得两者。
第 7 层负载平衡 - 是基于数据的应用程序级别平衡，因此网络数据包的数据内容需要路由到端点。 您可以通过在应用程序级别、服务级别或内核级别实现路由来实现容量（更多用户）。
可扩展性 - 我假设您的内存、CPU 资源和网络带宽都已耗尽。
应用程序级别 - 您的应用程序逻辑接收应用程序数据包并相应地进行路由。
应用
服务级别 - 您的系统框架（某种前端服务）接收数据包并通过模块 - 执行路由（考虑自定义 apache 模块，甚至网络驱动程序模块 - 就像编写网络过滤器）
内核级别 - 执行网络数据包级别的路由。
你离金属越近，你的反应就越好。 我建议预先使用专用的 Linux 服务器来执行路由——使用本地服务器，而不是虚拟服务器。 对 WAN 使用多个或组合的网络适配器，对每个端点使用专用适配器（一个+ WAN，每个连接的应用程序服务器各一个）
如果响应时间很重要，那么您需要内核/管理程序状态解决方案，它将为您节省很少有上下文切换，但请注意，您需要不惜一切代价限制跳数，并且可以通过更少、更大的机器更好地提供服务，并且您的可扩展性将始终受到限制。 使用 KTCPVS 存在风险，它相当旧且不主动更新。 如果您认为它非常适合您，否则请考虑编写类似于网络过滤器的东西，只要它在系统状态下运行即可。
如果容量很重要但响应时间是次要的，请实现一个用 C++ 构建的定制高速套接字开关，在问题/用户状态下运行。 它是最容易维护的，并且提供最好的可扩展性。
您将需要构建一些原型来找出最适合您需求的原型。
最后的想法 -
在执行上述任何操作之前，首先确保您已经优化了游戏设计。 这个大家可能都知道了，我把它列在这里，方便大家。
(a) 消息应该适合一个网络数据包，对于大多数家庭路由器来说小于 1500 字节
(b) 尝试适应游戏客户端而不是服务器中的路由逻辑。 只需将包含区域和 IP 地址的小表下载到客户端即可让您放弃上述所有操作。
(c) 尝试限制客户端的区域可见性，他们应该只知道自己的区域和相邻区域（如果您实施上面的 b 点）
希望这有帮助，抱歉我不能更具体地介绍 KTCPVS。
In order to address your question we need to understand whether you need volume or response, but it is difficult to get both at the same time.
Layer 7 load balancing - is data based application level balancing, so the data content of the network packet needs to be routed to an end-point. You can achieve volume (more users) by implementing routing at the application level, service level or kernel level.
Scalability - I assume you are running out of memory, CPU resources and network bandwidth.
Application level - your application logic receives an application packet and routes accordingly.
Service level - your system framework (front end service of some kind) receives the packet and through a module - performs the routing (think of custom apache module, even network driver modules - like writing a network filter)
Kernel level - Performs routing at network packet level.
The closer you move to the metal, the better your response will be. I suggest using dedicated linux server up-front to perform the routing - go native, not virtual. Use multiple or teamed network adapters for the WAN and a dedicated adapter for each end-point (one+ wan, one each for each connected app server)
If response time is important then you need a kernel/supervisor state solution, it will save you a few context switches but be aware that you need to limit hops at all costs and could better be served by fewer, larger machines and your scalability will always be limited. There is a risk in using KTCPVS, it is quite old and not actively updated. If you judge that it works for you great, otherwise consider writing something akin to a network filter as long as it runs in system state.
If volume is important but response time is secondary, implement a custom built high-speed socket switch built in C++ running in problem/user state. It is the easiest to maintain and will offer the best scalability.
You will need to build some prototypes to figure out what suits your needs best.
Final thoughts -
Before doing any of the above first ensure that you have optimized your game design. You may know most of this, I list it here for the benefit of all.
(a) messages should fit comfortably within one network packet, less than 1500 bytes for most home routers
(b) Try to fit the logic of the routing in your game client instead of your servers. A simple download of a small table with zones and IP addresses to a client will allow you to forego all of the above.
(c) Try to limit zone visibility by to the clients, they should know about their zones and adjacent zones only (if you implement the point b above)
Hope this helps, sorry I cannot be more specific regarding KTCPVS.
为什么要将分配逻辑移至负载均衡器？ 它是一个不是免费的且可能损坏的组件。 您的客户似乎非常清楚他们所在的区域。他们似乎可以很好地连接到
zone.example.com 。 然后，您可以在 DNS 级别处理负载平衡。Why are you moving the distribution logic to the loadbalancer? It's a component that's not free and can break. It seems your clients are quite aware of which zone they're in. It seems they could very well connect to
zone<n>.example.com. You'd then handle loadbalancing at DNS level.您还没有指定瓶颈在哪里。 网络流量？ 磁盘IO？ CPU 周期？
假设您指的是第 7 层负载均衡器并且没有足够的 CPU 能力，我认为 LVS 不是最佳选择。 我已经使用 LVS 完成了 Web 服务器负载平衡，它工作起来很简单，并不复杂。
但我认为以这种方式对 MMORP 进行负载平衡需要在 LVS 中添加大量额外代码，使用分布在某个多核服务器上的多线程应用程序进行负载平衡可能会更容易。 但这并不是完全可扩展的，这只会让您达到 16 个核心，而不会导致成本增加。
You haven't specified where the bottleneck is. Network Traffic? Disk IO? CPU Cycles?
Assuming you mean a layer 7 load balancer and don't have enough CPU power, I think LVS ist not the optimal choice. I have done Web Server load balancing with LVS, which works straightforward and isn't exactly complicated.
But I think load balancing an MMORP this way needs considerable amounts of additional code in LVS, it might be easier to do the load balancing with a multithreaded application distributed over some multicore server. But this isn't fully scalable, this only gets you to 16 cores without prohibitve cost increase.
此类事件中最大的问题是当玩家靠近边界时会发生什么。 显然，他们需要能够看到彼此并进行交互，但他们位于不同的服务器上。 因此，您需要一些非常奇特的服务器间通信，有时只是将消息复制到两台服务器。 当有人接近“角落”时，情况会变得更加复杂，然后你必须处理 4 个服务器！
大型多人游戏开发一书有一章“共享服务器边界的陷阱”，详细介绍了这个问题。
我之前没有听说过Linux虚拟服务器，所以我不明白它是如何适合的。 我认为您的实际服务器应用程序需要支持这种特定于游戏的负载平衡，而不是尝试运行集群并假设它将自动知道如何拆分您的应用程序（但它不会）。 如果我是你，我会编写服务器程序来处理自己的一块土地，并且它应该连接到它周围的一块土地，然后设计一个服务器到服务器的协议来传递这些消息（“这里来了一个玩家，我要开始告诉你有关他的信息！”“确保告诉我有关我们边界附近的消息”，“好吧，该玩家离开了我的领土并进入了你的领土，这是他的详细数据”，等等） 。 我认为这比仅仅运行不同风格的 Linux 并假设您将获得自动负载平衡要复杂一些。
The biggest issue in something like this is what happens when players are near a boundary. Obviously they need to be able to see and interact with each other, but they're on separate servers. So you need some pretty fancy inter-server communication, sometimes just duplicating messages to both servers. It can get even more complicated when someone is near a "corner", and then you have to deal with 4 servers!
The book Massively Multiplayer Game Development has a chapter on "The Pitfalls of Shared Server Boundaries" which covers this issue in detail.
I haven't heard of Linux Virtual Server before now, so I don't understand how it fits. I think your actual server application needs to support this game-specific load balancing, rather than trying to run a cluster and assuming that it will automatically know how to split up your application (which it won't). If I were you, I would write the server program to handle its own piece of land, and it should connect to the pieces of land around it, and then design a server-to-server protocol for the passing of these messages ("here comes a player, I'm going to start telling you about him!" "make sure to tell me about messages near our boundary", "okay the player is out of my territory and into yours, here's his detailed data", etc). I think it's a bit more complicated than just running a different flavor of Linux and assuming you'll get automatic load balancing.