内核如何处理新文件的创建
我希望了解当用户/应用程序尝试在目录中创建文件时内核的工作方式。
背景 - 我们有一个 java 应用程序，它通过 JMS 使用消息，对其进行处理，然后将 XML 写入出站队列+本地目录。昨天，我们观察到写入目录时出现异常延迟。在“ls|wc -l”上，我们发现了超过 300,000 个文件。对进程进行了快速跟踪，发现它充满了互斥调用（strace 中超过 3/4 的调用是互斥）。
所以我认为创建新文件需要时间，因为系统每次都必须在 300,000 个文件中检查某些内容（例如文件名称以确保可以创建具有特定名称的新文件），然后创建一个文件。
我清除了目录，应用程序恢复到正常服务级别。
我的问题
- 我的分析正确吗（似乎是因为应用程序在清除后开始正常工作）？
- 更重要的是，当您尝试在目录中创建新文件时，内核如何工作。
- 互斥调用次数异常是否可以归因于目录中文件数量过多？
非常感谢 J
I wish to understand the way kernel works when a user/app tries to create a file in a directorty.
The background - We have a java applicaiton which consumes messages over JMS, processes it and then writes the XML to an outbound queue+a local directory. Yesterday we obeserved unsual delays in writing to the directory. On 'ls|wc -l' we found >300,000 files in there. Did a quick strace on the process and found it full of mutex calls (More than 3/4 calls in the strace were mutex).
So i thought that new file creation is taking time becasue the system has to every time check certain things (e.g name of files to make sure that the new file with a specific name can be created) amongst 300,000 files and then create a file.
I cleared the directory and the applicaiton resumed to normal service levels.
My questions
- Was my analysis correct (It seems cuz the app started working fine after a clear down)?
- More imporatant, how does the kernel work when you try to creat a new file in directory.
- Can the abnormal number of mutex calls be attributed to the high number of files in the directory?
Many thanks
J
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
请阅读有关 Linux 文件系统、i 节点和 d 节点的信息。
的
文件系统被组织成固定 大小的块。如果您的目录相对较小，则它适合直接块，并且速度很快。如果您的目录不是太大，它适合直接块和一些间接块，并且仍然相当快。如果您的目录变得太大，它会溢出到双间接块并且变得很慢。
实际大小取决于文件系统和内核配置。
经验法则是将目录保持在 12 个块以下，具体取决于块大小。许多系统使用8K块；快速目录小于 98,304 字节。
文件条目的大小约为 16*4 字节 (IIRC)，因此计划每个目录的文件数不超过 1500 个作为实际上限。
Please read about the Linux Filesystem, i-nodes and d-nodes.
http://en.wikipedia.org/wiki/Inode_pointer_structure
The file system is organized into fixed-sized blocks. If your directory is relatively small, it fits in the direct blocks and things are fast. If your directory is not too big, it fits in the direct blocks and some indirect blocks, and is still reasonably fast. If your directory becomes too big, it spills into double indirect blocks and becomes slow.
Actual sizes depend on file system and kernel configuration.
Rule of thumb is to keep the directory under 12 blocks, depending on your block size. Many systems use 8K blocks; a fast directory is under 98,304 bytes.
A file entry is something like 16*4 bytes in size (IIRC), so plan on no more than 1500 files per directory as a practical upper limit.
具有大量条目的目录通常很慢 - 慢多少取决于底层文件系统。
常见的解决方案是创建目录层次结构，因此每个目录只有几百个条目。
Directories with large numbers of entries are often slow - how slow depends on the underlying filesystem.
The common solution is to create a hierarchy of directories, so each dir only has a few hundred entries.
互斥系统调用是应用程序（可能是 JVM 或 Java 库中的某些东西）进行互斥调用的结果。
您不会通过 strace 看到内核内部的同步，因为这仅检查系统调用本身。
如果您使用使用目录索引的文件系统，则包含大量文件的目录不应变得低效；现在大多数都这样做（ext3 可以选择这样做，但现在通常会启用）。
非索引目录（比如在糟糕的旧文件系统上使用的目录 - ext2、vfat 等）在处理大量文件时会变得非常糟糕，并且您会看到“打开”系统调用需要更长的时间。
Mutex system calls are a result of the application (probably something in the JVM or the Java libraries) making mutex calls.
Synchronisation internal to the kernel you will not see via strace, as this only examines system calls themselves.
A directory with lots of files should not become inefficient if you are using a filesystem which uses directory indexes; most now do (ext3 does optionally but it's normally enabled nowadays).
Non-indexed directories (like those used on the bad old filesystems - ext2, vfat etc) get really bad with lots of files, and you'll see the "open" system call taking a lot longer.