如何从 Kestrel 队列中有效获取数据
由于某种原因，我们计划在我们的项目中使用 Kestrel 队列。我们做了一些恶魔，主要问题是如何在低CPU利用率的情况下有效地从队列中获取数据。我们实现的获取方式是，如果从队列中获取数据失败超过 5 次，我们将线程休眠 100 毫秒以降低 CPU 利用率。
while (running) {
try {
LoginLogQueueEntry data = kestrelQueue.fetch();
if (null != data && data.isLegal()) {
entryCacheList.add(data); //add the data to the local caceh
resetStatus();
} else {
failedCount++;
//if there is no data in the kestrel and the local cache is not empty, insert the data into mysql database
if (failedCount == 1 && !entryCacheList.isEmpty()) {
resetStatus();
insertLogList(entryCacheList); // insert current data into database
entryCacheList.clear(); //empty local cache
}
if (failedCount >= 5 && entryCacheList.isEmpty()) {
//fail 5 times. Sleep current thread.
failedCount = 0;
Thread.sleep((sleepTime + MIN_SLEEP_TIME) % MAX_SLEEP_TIME);
}
}
//Insert 1000 rows once
if (entryCacheList.size() >= 1000) {
insertLogList(entryCacheList);
entryCacheList.clear();
}
} catch (Exception e) {
logger.warn(e.getMessage());
}
还有其他好的方法吗？我认为最完美的方式是队列可以通知工作人员我们获取了数据并获取它们。
For some reason we plan to use kestrel queue in our project. We do some demons, the main problem is how to to fetch data from queue with low CPU utilization and effectively. The way we implemented to fetch is if we failed to fetch data from queue more than 5 times, we sleep the thread 100ms to reduce the CPU utilization.
while (running) {
try {
LoginLogQueueEntry data = kestrelQueue.fetch();
if (null != data && data.isLegal()) {
entryCacheList.add(data); //add the data to the local caceh
resetStatus();
} else {
failedCount++;
//if there is no data in the kestrel and the local cache is not empty, insert the data into mysql database
if (failedCount == 1 && !entryCacheList.isEmpty()) {
resetStatus();
insertLogList(entryCacheList); // insert current data into database
entryCacheList.clear(); //empty local cache
}
if (failedCount >= 5 && entryCacheList.isEmpty()) {
//fail 5 times. Sleep current thread.
failedCount = 0;
Thread.sleep((sleepTime + MIN_SLEEP_TIME) % MAX_SLEEP_TIME);
}
}
//Insert 1000 rows once
if (entryCacheList.size() >= 1000) {
insertLogList(entryCacheList);
entryCacheList.clear();
}
} catch (Exception e) {
logger.warn(e.getMessage());
}
Is there any other good way to do so? The perfect the way i think is the queue can notice to the worker that we got data and fetch them .
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
请参阅 http://robey.lag 中的“阻止获取”部分.net/2008/11/27/scarling-to-kestrel.html
See the "Blocking Fetches" section at http://robey.lag.net/2008/11/27/scarling-to-kestrel.html
此处的“Memcache 命令”下描述了阻止读取： https://github .com/robey/kestrel/blob/master/docs/guide.md
您可以将选项标志添加到 get 命令中，方法是用斜杠分隔选项标志，以便从“作业”队列中获取项目，等待一秒钟：
如果一秒钟内队列中没有任何内容显示，您将得到相同的空响应，仅比现在得到的晚一秒钟。 :)
执行此操作时，调整响应超时非常重要。如果您使用超时为 1 秒的阻塞读取，但客户端库的响应超时为 500 毫秒，则库将在阻塞读取完成之前与服务器断开连接。因此，请确保响应超时大于您在读取请求中使用的超时。
Blocking reads are described here, under "Memcache commands": https://github.com/robey/kestrel/blob/master/docs/guide.md
You can add option flags to a get command by separating them with slashes, so to fetch an item from the "jobs" queue, waiting up to one second:
If nothing shows up on the queue in one second, you'll get the same empty response, just one second later than you're getting it now. :)
It's important to tune your response timeout when you do this. If you use a blocking read with a timeout of one second, but your client library's response timeout is 500 milliseconds, the library will disconnect from the server before the blocking read is finished. So make sure the response timeout is greater than the timeout you're using in the read request.
您需要使用阻塞 get。我无法找到 API 文档，但我发现一篇文章表明这在 kestrel 中是可能的。
You need to use a blocking get. I couldn't track down the API docs, but I found an article suggesting that it's possible in kestrel.