为什么membase服务器响应时间这么慢?
我有一个问题,membase 在我的环境中速度非常慢。 我在 Rails 2.3.10 ruby 1.8.7 上运行多个生产服务器(Passenger)。 这些服务器与集群中的 2 台 membase 机器通信。
每台 membase 机器都有 64G 内存和附加的 100G EBS,1G 交换区。
我的问题是 membase 的响应时间非常慢,实际上是整个应用程序生命周期中最慢的部分。
我的问题是:为什么?
我使用的 Rails gem 是 memcache-northscale。 membase 服务器是 1.7.1(最新)。
服务器每秒执行 2K-7K 操作(对于集群)
membase(基于 NewRelic)的响应时间平均为 250 毫秒,这是巨大且不合理的。
有谁知道为什么会发生这种情况? 这次我能做些什么来改善呢?
I have a problem that membase is being very slow on my environment.
I am running several production servers (Passenger) on rails 2.3.10 ruby 1.8.7.
Those servers communicate with 2 membase machines in a cluster.
the membase machines each have 64G of memory and a100g EBS attached to them, 1G swap.
My problem is that membase is being VERY slow in response time and is actually the slowest part right now in all of the application lifecycle.
my question is: Why?
the rails gem I am using is memcache-northscale.
the membase server is 1.7.1 (latest).
The server is doing between 2K-7K ops per second (for the cluster)
The response time from membase (based on NewRelic) is 250ms in average which is HUGE and unreasonable.
Does anybody know why is this happening?
What can I do inorder to improve this time?
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(2)
很难立即根据手头的数据做出判断,但我认为您可能希望深入研究一些内容,以缩小问题的范围。
首先,您的 membase 统计数据是否显示大量后台提取?这是 Web UI 统计信息中的“每秒磁盘读取次数”。如果是这样,这可能是导致较高延迟的罪魁祸首。
您可以在 手册,特别是有关统计和集群设计注意事项的部分。
其次,您报告的平均时间为 250 毫秒。这是滑动平均值还是整体?您是否有最大 90 或最大 99 延迟之类的东西?当大多数请求(例如,来自 RAM 的不需要磁盘提取的请求)实际上相当快时,一些外围磁盘提取可能会给您带来很大的平均值。
您的系统是否分布在整个可用区域?您使用什么类型的实例?客户端和服务器是否位于同一个 Amazon AWS 区域?我怀疑第一个答案可能是“是”,这意味着使用最近测量的 xlarge 实例时大约有 1.5 毫秒的开销。如果您在给定方法中同步和串行执行大量提取,这可能很重要。
我预计这一切都在一个区域,但值得仔细检查,因为这些延迟听起来像 WAN 延迟。
最后,还有一个更新的 Ruby gem,向后兼容 Fauna。 Couchbase, Inc. 一直致力于恢复 Fauna 上游。如果可能的话,您可能想尝试这里引用的 gem:
http://www.couchbase.org/code/couchbase/ruby/2.0.0
It's hard to immediately say with the data at hand, but I think I have a few things you may wish to dig into to narrow down where the issue may be.
First of all, do your stats with membase show a significant number of background fetches? This is in the Web UI statistics for "disk reads per second". If so, that's the likely culprit for the higher latencies.
You can read more about the statistics and sizing in the manual, particularly the sections on statistics and cluster design considerations.
Second, you're reporting 250ms on average. Is this a sliding average, or overall? Do you have something like max 90th or max 99th latencies? Some outlying disk fetches can give you a large average, when most requests (for example, those from RAM that don't need disk fetches) are actually quite speedy.
Are your systems spread throughout availability zones? What kind of instances are you using? Are the clients and servers in the same Amazon AWS region? I suspect the answer may be "yes" to the first, which means about 1.5ms overhead when using xlarge instances from recent measurements. This can matter if you're doing a lot of fetches synchronously and in serial in a given method.
I expect it's all in one region, but it's worth double checking since those latencies sound like WAN latencies.
Finally, there is an updated Ruby gem, backwards compatible with Fauna. Couchbase, Inc. has been working to add back to Fauna upstream. If possible, you may want to try the gem referenced here:
http://www.couchbase.org/code/couchbase/ruby/2.0.0
您还需要考虑在客户端运行 Moxi。通过访问Membase,你需要通过一个代理(称为Moxi)。默认情况下,它安装在服务器上,这意味着您可能会向实际上没有密钥的服务器之一发出请求。 Moxi 会去获取它...但是这样你的网络流量就会增加一倍。
在客户端安装 Moxi 将消除这种额外的网络流量: http://www.couchbase .org/wiki/display/membase/Moxi
佩里
You will also want to look at running Moxi on the client-side. By accessing Membase, you need to go through a proxy (called Moxi). By default, it's installed on the server which means you might make a request to one of the servers that doesn't actually have the key. Moxi will go get it...but then you're doubling the network traffic.
Installing Moxi on the client-side will eliminate this extra network traffic: http://www.couchbase.org/wiki/display/membase/Moxi
Perry