生产环境中随机长数据库查询/Memcache 获取
我无法诊断 ubuntu scalr/ec2 生产环境中遇到的问题。
问题显然是随机的,数据库查询和/或内存缓存查询将花费比应有的更长的时间。 我见过一个简单的 select 语句需要 130 毫秒,或者一个 Memcache 获取需要 65 毫秒! 每个请求可能会发生几次,导致某些请求花费的时间是应有的时间的两倍。
为了诊断问题,我编写了一个非常简单的脚本,它将仅连接到 MySql 服务器并运行查询。
require 'mysql'
mysql = Mysql.init
mysql.real_connect('', '', '', '')
max = 0
100.times do
start = Time.now
mysql.query('select * from navigables limit 1')
stop = Time.now
total = stop - start
max = total if total > max
end
puts "Max Time: #{max * 1000}"
mysql.close
该脚本始终返回非常高的最大时间,因此我消除了任何 Rails 作为问题根源。 我也在 Python 中写了同样的东西来消除 Ruby。 事实上,Python 也花费了大量的时间!
MySql 和 Memcache 都在自己的盒子上,因此我考虑了网络延迟,但观察 ping
和 traceroute
ing 看起来很正常。
另外,在相应的计算机上运行查询/获取会返回预期时间,并且我在登台计算机上运行相同版本的 gems,没有出现此问题。
我真的很难过这个……对我可以尝试诊断这个问题有什么想法吗? 谢谢
I'm having trouble diagnosing a problem I'm having on my ubuntu scalr/ec2 production environment.
The trouble is apparently randomly, database queries and/or memcache queries will take MUCH longer than they should. I've seen a simple select statement take 130ms or a Memcache fetch take 65ms! It can happen a handful of times per request, causing some requests to take twice as long as they should.
To diagnose the problem, I wrote a very simple script which will just connect to the MySql server and run a query.
require 'mysql'
mysql = Mysql.init
mysql.real_connect('', '', '', '')
max = 0
100.times do
start = Time.now
mysql.query('select * from navigables limit 1')
stop = Time.now
total = stop - start
max = total if total > max
end
puts "Max Time: #{max * 1000}"
mysql.close
This script consistently returned a really high max time, so I eliminated any Rails as the source of the problem. I also wrote the same thing in Python to eliminate Ruby. And indeed the Python one took inordinate amounts of time as well!
Both MySql and Memcache are on their own boxes, so I considered network latency, but watching ping
s and traceroute
ing look normal.
Also running the queries/fetches on the respective machines returns expected times, and I'm running the same version gems on my staging machine without this issue.
I'm really stumped on this one... any thoughts on something I could try to diagnose this?
Thanks
如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱,如果其他用户或者作者回复了您的评论,将不能在第一时间通知您!
发布评论
评论(2)
我唯一的想法是它可能是磁盘?
My only thought is that it might be disk?
Mysql 使用查询缓存来存储 SELECT 及其结果。 这可以解释你在连续选择中获得的恒定速度。 尝试 EXPLAIN-inig 查询以查看是否正在使用索引。
我不明白为什么内存缓存会成为问题(除非它崩溃并重新启动?)。 检查您的服务器日志是否有可疑的服务故障。
Mysql uses query cache to store SELECT together with its result. That could explain the constant speed you are getting in continous selecting. Try EXPLAIN-inig the query to see if you are using indexes.
I don't see why memcache would be a problem (unles it's crashing and restarting?). Check your server logs for suspicious service failures.