C# 如何在不使用大量内存的情况下递归循环一大组文件夹和文件

发布于 2024-10-04 08:30:37 字数 341 浏览 4 评论 0原文

我想索引所有音乐文件并将它们存储在数据库中。我有一个从音乐驱动器的根目录开始反复调用的函数。

即

start > ReadFiles(C:\music\);

ReadFiles(path){
   foreach(file)
      save to index;

   foreach(directory)
      ReadFiles(directory);
}

这工作正常，但是在运行程序时，使用的内存量不断增加......最后我的系统内存不足。

有没有人有更好的方法，不需要 4GB RAM 来完成这个任务？

最好的问候，泰斯

原文

I want to index all my music files and store them in a database.
I have this function that i call recusively, starting from the root of my music drive.

i.e.

start > ReadFiles(C:\music\);

ReadFiles(path){
   foreach(file)
      save to index;

   foreach(directory)
      ReadFiles(directory);
}

This works fine, but while running the program the amount of memory that is used grows and grows and.. finally my system runs out of memory.

Does anyone have a better approach that doesnt need 4GB of RAM to complete this task?

Best Regards, Tys

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

空袭的梦i 2024-10-11 08:30:37

Alxandr 基于队列的解决方案应该可以正常工作。

如果您使用的是 .NET 4.0，您还可以利用新的 Directory.EnumerateFiles 方法，该方法会延迟枚举文件，而无需将它们全部加载到内存中：

void ReadFiles(string path)
{
    IEnumerable<string> files =
        Directory.EnumerateFiles(
            path,
            "*",
            SearchOption.AllDirectories); // search recursively

    foreach(string file in files)
        SaveToIndex(file);
}

Alxandr's queue based solution should work fine.

If you're using .NET 4.0, you could also take advantage of the new Directory.EnumerateFiles method, which enumerates files lazily, without loading them all in memory:

void ReadFiles(string path)
{
    IEnumerable<string> files =
        Directory.EnumerateFiles(
            path,
            "*",
            SearchOption.AllDirectories); // search recursively

    foreach(string file in files)
        SaveToIndex(file);
}

回复收藏 0 原文

孤单情人 2024-10-11 08:30:37

您是否检查了除根目录之外的每个目录中显示的 . 和 .. 条目？

如果你不跳过这些，你就会陷入无限循环。

回复收藏 0 原文

别想她 2024-10-11 08:30:37

您可以将其实现为队列。我认为（但我不确定）这会节省内存。至少它会释放你的堆栈。每当您找到一个文件夹时，您都会将其添加到队列中，每当您找到一个文件时，您都会读取它。这可以防止递归。

像这样的东西：

Queue<string> dirs = new Queue<string>();
dirs.Enqueue("basedir");
while(dirs.Count > 0) {
    foreach(directory)
        dirs.Enqueue(directory);
    ReadFiles();
}

You can implement this as a queue. I think (but I'm not sure) that this will save memory. At least it will free up your stack. Whenever you find a folder you add it to the queue, and whenever you find a file you just read it. This prevents recursion.

Something like this:

Queue<string> dirs = new Queue<string>();
dirs.Enqueue("basedir");
while(dirs.Count > 0) {
    foreach(directory)
        dirs.Enqueue(directory);
    ReadFiles();
}

回复收藏 0 原文

微暖i 2024-10-11 08:30:37

但请注意，如果您无权访问文件或路径太长或发生其他异常，EnumerateFiles() 将停止运行。这就是我目前用来解决这些问题的方法：

public static List<string> getFiles(string path, List<string> files)
{
    IEnumerable<string> fileInfo = null;
    IEnumerable<string> folderInfo = null;
    try
    {
        fileInfo = Directory.EnumerateFiles(str);
    }
    catch
    {

    }
    if (fileInfo != null)
    {
        files.AddRange(fileInfo);
        //recurse through the subfolders
        fileInfo = Directory.EnumerateDirectories(str);
        foreach (string s in folderInfo)
        {
            try
            {
                getFiles(s, files);
            }
            catch
            {

            }
        }
    }
    return files;
}

使用示例：

List<string> files = new List<string>();
files = folder.getFiles(path, files);

我的解决方案基于此页面上的代码： http://msdn.microsoft.com/en-us/library/vstudio/bb513869.aspx。

更新：可以在 http://social.msdn.microsoft.com/Forums/vstudio/en-US/ae61e5a6-97f9-4eaa-9f1a-856541c6dcce/directorygetfiles -gives-me-access-denied?forum=csharpgeneral。使用 Stack 对我来说是新的（我什至不知道它的存在），但该方法似乎有效。至少它列出了我的 C 和 D 分区上的所有文件，没有错误。

Beware, though, that EnumerateFiles() will stop running if you don't have access to a file or if a path is too long or if some other exception occurs. This is what I use for the moment to solve those problems:

public static List<string> getFiles(string path, List<string> files)
{
    IEnumerable<string> fileInfo = null;
    IEnumerable<string> folderInfo = null;
    try
    {
        fileInfo = Directory.EnumerateFiles(str);
    }
    catch
    {

    }
    if (fileInfo != null)
    {
        files.AddRange(fileInfo);
        //recurse through the subfolders
        fileInfo = Directory.EnumerateDirectories(str);
        foreach (string s in folderInfo)
        {
            try
            {
                getFiles(s, files);
            }
            catch
            {

            }
        }
    }
    return files;
}

Example use:

List<string> files = new List<string>();
files = folder.getFiles(path, files);

My solution is based on the code at this page: http://msdn.microsoft.com/en-us/library/vstudio/bb513869.aspx.

Update: A MUCH faster method to get files recursively can be found at http://social.msdn.microsoft.com/Forums/vstudio/en-US/ae61e5a6-97f9-4eaa-9f1a-856541c6dcce/directorygetfiles-gives-me-access-denied?forum=csharpgeneral. Using Stack is new to me (I didn't even know it existed), but the method seems to work. At least it listed all files on my C and D partition with no errors.

回复收藏 0 原文