从驱动器读取大文件的部分内容

发布于 2024-10-20 15:43:18 字数 260 浏览 12 评论 0原文

我正在 C# 中处理大文件（最多可达可用内存的 20%-40%），并且一次只需要将文件的一小部分加载到内存中（例如文件的 1-2%））。我认为使用 FileStream 将是最好的选择，但我不知道。我需要给出一个起点（以字节为单位）和长度（以字节为单位）并将该区域复制到 byte[] 中。对文件的访问可能需要在线程之间共享，并且将在文件中的随机位置进行（非线性访问）。我也需要它快点。

该项目已经有 unsafe 方法，因此请随意从 C# 更危险的方面提出建议

原文

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

甩你一脸翔 2024-10-27 15:43:18

FileStream 将允许您查找所需文件的部分，没有问题。这是在 C# 中执行此操作的推荐方法，而且速度很快。

线程之间共享：您需要创建一个锁，以防止其他线程在您尝试读取文件流时更改它的位置。最简单的方法是：

//  This really needs to be a member-level variable;
private static readonly object fsLock = new object();

//  Instantiate this in a static constructor or initialize() method
private static FileStream fs = new FileStream("myFile.txt", FileMode.Open);


public string ReadFile(int fileOffset) {

    byte[] buffer = new byte[bufferSize];

    int arrayOffset = 0;

    lock (fsLock) {
        fs.Seek(fileOffset, SeekOrigin.Begin);

        int numBytesRead = fs.Read(bytes, arrayOffset , bufferSize);

        //  Typically used if you're in a loop, reading blocks at a time
        arrayOffset += numBytesRead;
    }

    // Do what you want to the byte array and return it

}

根据需要添加 try..catch 语句和其他代码。在您访问此 FileStream 的任何地方，请在成员级变量 fsLock 上放置一个锁...这将阻止其他方法在您尝试读取时读取/操作文件指针。

就速度而言，我想您会发现您受到磁盘访问速度的限制，而不是代码。

您必须考虑有关多线程文件访问的所有问题......谁初始化/打开文件，谁关闭它，等等。有很多基础知识需要涵盖。

A FileStream will allow you to seek to the portion of the file you want, no problem. It's the recommended way to do it in C#, and it's fast.

Sharing between threads: You will need to create a lock to prevent other threads from changing the FileStream position while you're trying to read from it. The simplest way to do this:

//  This really needs to be a member-level variable;
private static readonly object fsLock = new object();

//  Instantiate this in a static constructor or initialize() method
private static FileStream fs = new FileStream("myFile.txt", FileMode.Open);


public string ReadFile(int fileOffset) {

    byte[] buffer = new byte[bufferSize];

    int arrayOffset = 0;

    lock (fsLock) {
        fs.Seek(fileOffset, SeekOrigin.Begin);

        int numBytesRead = fs.Read(bytes, arrayOffset , bufferSize);

        //  Typically used if you're in a loop, reading blocks at a time
        arrayOffset += numBytesRead;
    }

    // Do what you want to the byte array and return it

}

Add try..catch statements and other code as necessary. Everywhere you access this FileStream, put a lock on the member-level variable fsLock... this will keep other methods from reading/manipulating the file pointer while you're trying to read.

Speed-wise, I think you'll find you're limited by disk access speeds, not code.

You'll have to think through all the issues about multi-threaded file access... who intializes/opens the file, who closes it, etc. There's a lot of ground to cover.

回复收藏 0 原文