Linux 异步 IO 与 libaio 性能问题
我正在尝试使用 Linux libaio 来优化服务器应用程序中的 IO 性能。我相信我已经做了所有必要的事情（使用 O_DIRECT，将缓冲区与内存页对齐......）。我期望对 io_submit 的调用立即返回，但一个简单的测试显示，在我的 Core i7 笔记本电脑上，它实际上需要大约 80 微秒的时间才能返回。是我的期望太高还是我的测试程序有问题？ （用 g++ --std=c++0x -laio 编译）
#include <unistd.h>
#include <fcntl.h>
#include <libaio.h>
#include <errno.h>
#include <cstdlib>
#include <cstdio>
#include <iostream>
#include <chrono>
// Open the file for write, return the file descriptor
int open_write(char const* file)
{
int fd = open(file, O_DIRECT|O_CREAT|O_WRONLY, S_IRWXU|S_IRWXG|S_IROTH);
if (fd < 0) {
perror("open_write");
exit(1);
}
}
// Make a buffer of _size_ byte, fill with 'a', return the buffer, it should be aligned to memory page
void* make_write_buffer(size_t size)
{
void* buf = 0;
int ret = posix_memalign((void**)&buf, sysconf(_SC_PAGESIZE), size);
if (ret < 0 || buf == 0) {
perror("make_write_buffer");
exit(1);
}
memset(buf, 'a', size);
return buf;
}
int main (int argc, char *argv[])
{
static const size_t SIZE = 16 * 1024;
// Prepare file and buffer to write
int write_fd = open_write("test.dat");
void* buf = make_write_buffer(SIZE);
// Prepare aio
io_context_t ctx;
memset(&ctx, 0, sizeof(ctx));
const int maxEvents = 32;
io_setup(maxEvents, &ctx);
iocb *iocbpp = new iocb;
io_prep_pwrite(iocbpp, write_fd, buf, SIZE, 0);
using namespace std::chrono;
// Submit aio task
auto start = monotonic_clock::now();
int status = io_submit(ctx, 1, &iocbpp);
if (status < 0) {
errno = -status;
perror("io_submit");
exit(1);
}
auto dur = duration_cast<microseconds>(monotonic_clock::now() - start);
std::cout << "io_submit takes: " << dur.count() << " microseconds." << std::endl;
io_event events[10];
int n = io_getevents(ctx, 1, 10, events, NULL);
close(write_fd);
io_destroy(ctx);
delete iocbpp;
free(buf);
return 0;
}
I'm trying Linux libaio for optimized IO performance in server application. I believe I've done everything necessary (using O_DIRECT, align buffer with memory page...). I'm expecting the call to io_submit returns immediately, but a simple test showing it actually takes something around 80 micro seconds to return on my core i7 laptop. Am I expecting too much or there is something wrong with my test program? (compiled with g++ --std=c++0x -laio )
#include <unistd.h>
#include <fcntl.h>
#include <libaio.h>
#include <errno.h>
#include <cstdlib>
#include <cstdio>
#include <iostream>
#include <chrono>
// Open the file for write, return the file descriptor
int open_write(char const* file)
{
int fd = open(file, O_DIRECT|O_CREAT|O_WRONLY, S_IRWXU|S_IRWXG|S_IROTH);
if (fd < 0) {
perror("open_write");
exit(1);
}
}
// Make a buffer of _size_ byte, fill with 'a', return the buffer, it should be aligned to memory page
void* make_write_buffer(size_t size)
{
void* buf = 0;
int ret = posix_memalign((void**)&buf, sysconf(_SC_PAGESIZE), size);
if (ret < 0 || buf == 0) {
perror("make_write_buffer");
exit(1);
}
memset(buf, 'a', size);
return buf;
}
int main (int argc, char *argv[])
{
static const size_t SIZE = 16 * 1024;
// Prepare file and buffer to write
int write_fd = open_write("test.dat");
void* buf = make_write_buffer(SIZE);
// Prepare aio
io_context_t ctx;
memset(&ctx, 0, sizeof(ctx));
const int maxEvents = 32;
io_setup(maxEvents, &ctx);
iocb *iocbpp = new iocb;
io_prep_pwrite(iocbpp, write_fd, buf, SIZE, 0);
using namespace std::chrono;
// Submit aio task
auto start = monotonic_clock::now();
int status = io_submit(ctx, 1, &iocbpp);
if (status < 0) {
errno = -status;
perror("io_submit");
exit(1);
}
auto dur = duration_cast<microseconds>(monotonic_clock::now() - start);
std::cout << "io_submit takes: " << dur.count() << " microseconds." << std::endl;
io_event events[10];
int n = io_getevents(ctx, 1, 10, events, NULL);
close(write_fd);
io_destroy(ctx);
delete iocbpp;
free(buf);
return 0;
}
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
简而言之：
io_submit阻塞，除了修复内核之外你无能为力。这是 2011 年 9 月 23 日的“阻止 io_submit”线程在 linux-aio 邮件列表上。
正如该线程中指出的，您可以尝试增加
/sys/block/xxx/queue/nr_requestsIn short:
io_submitblocks, and there's nothing you can do about it short of fixing the kernel.Here's a "blocking io_submit" thread from 2011-09-23 on the linux-aio mailing list.
As pointed out in that thread, you could try increasing
/sys/block/xxx/queue/nr_requestsio_getevents需要多长时间？如果大部分时间都花在 io_submit 上而不是 io_getevents 上，那么 io 实际上可能已经在 io_submit 期间执行。 （就我而言，我怀疑这是 ext4s 的错误...）您可以尝试的另一件事是使用
taskset将进程固定到核心？顺便说一句，您还可以使用 strace -T 获取这些计时。
编辑：我怀疑 ext4 是错误的，结果发现缺少 O_DIRECT 标志导致了同步行为。
How long does
io_geteventstake? If most of the time are spent in io_submit and not inio_geteventsthen the io may actually already get executed duringio_submit. (In my case I suspect that's ext4s fault...)Another thing you can try is pinning your process to a core using
taskset?Btw you can also get these timings using
strace -T.Edit: I was wrong in suspecting ext4, turns out the lack of O_DIRECT flag was causing the synchronous behaviour.
正如@Arvid在另一个答案中指出的，
io_submit()可以阻止。 有一个io_submit()可能阻塞的非详尽原因列表。 href="https://stackoverflow.com/questions/34572559/asynchronous-io-io-submit-latency-in-ubuntu-linux">异步IO io_submit延迟在 Ubuntu Linux 中，我猜测文件扩展写入正在 ext4 文件系统上执行，从而导致 io_submit() 在写入文件系统元数据时同步......As pointed out in the other answer by @Arvid,
io_submit()can block. There is a non-exhaustive list of reasons whyio_submit()can block over on asynchronous IO io_submit latency in Ubuntu Linux and here I'd guess that file extending writes are being performed on an ext4 filesystem thus causingio_submit()to go synchronous while filesystem metadata is written...