将物理磁盘转储到文件上并同时对其进行哈希处理
我创建了一个 WPF 应用程序，它将物理设备（硬盘、USB 大容量存储）读取到文件中。 HD 的速度可达 75-80 MB/s，USB 的速度可达 20-25 MB/s。我想动态添加图像的 MD5 和 SHA1 哈希值。基本上，我从磁盘读取 128 个扇区 (64Kb) 的块，使用两个算法对其进行哈希处理，然后将该块写入另一个文件。
嗯，这两个哈希函数似乎是一个巨大的瓶颈。 USB 速度降至 5 Mb/s，HD 速度降至 20 Mb/s。
所以我尝试将 hasing 代码放在另一个线程中。因此，一个线程读取块并将值放入 FIFO 列表（使用 List 制作）中。然后另一个线程弹出一个块并拥有它。 问题在于，从磁盘读取的线程比散列读取的速度更快，并且列表会呈指数增长。所以我将列表限制为 1024 个块。当第一个线程看到列表有 1024 个块或更多时，它会休眠直到达到 512 个……
这似乎在前几秒钟有效。我的 USB 速度为 19 Mb/s。但过了一会儿，它又会变慢。我想它填满了列表，并且正在等待它有一些空间......
也许我的 hasing 函数很慢？我把网上找到的第一个复制过来了... 如何提高申请速度？
谢谢
I've created a WPF application which reads a physical device (hard disk, usb mass storage) to a file. I get speeds up to 75-80 MB/s for HDs and 20-25 for USBs. I wanted to add MD5 and SHA1 hashing of the image on the fly. Basically I read a block of 128 sectors (64Kb) from the disk, I hash it using the two algos, and then I write the block to another file.
Well the two hashing functions seem to be a huge bottleneck. The speed went down to 5 Mb/s for Usb and 20 Mb/s for HD.
So I tought to put the hasing code in another thread. So one thread reads the blocks and puts the value in a FIFO list (made using List). Then another thread pops out a block and hases it.
The problem is that the thread reading from the disk is faster than the hashing one and the List would grow exponentially. So I put a limit on the list of 1024 blocks. When the first thread sees that the list has 1024 blocks or more, it sleeps until it gets to 512...
This seems to work for the first few seconds. I get 19 Mb/s for the Usb. But just after a bit, it tends to get slower again. I suppose it filled the list and it's waiting for it to have some space...
Maybe my hasing functions are slow? I copied the first one that I found on the web...
How can I enhance the speed of my application?
Thanks
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（3）
显然，如果有足够的时间，吞吐量将受到流程中较慢部分性能的限制。使 IO 与散列异步的优点在于您可以使两个进程一起运行，而不是您会神奇地比其中任何一个进程更快。
很难知道为什么你的散列如此慢，但是 .NET 包含 MD5 和 SHA 实现，因此你不需要编写自己的散列。
据推测，如果您有两个线程访问一个列表（队列或并发队列可能更好），那么您就对其进行了一些锁定。您确定没有在一个线程上长时间持有锁，从而导致另一个线程被阻塞吗？
理想情况下，您需要运行某种分析器，但您也许可以使用秒表和一些跟踪来弄清楚发生了什么。
Clearly, given enough time, the throughput is going to be limited at the performance of whichever is the slower part of the process. The advantage of making the IO asynchronous to the hashing is just that you can keep both processes going together, not that you'll magically be faster than either of them individually.
It's hard to know why your hashing is so slow, but .NET contains both MD5 and SHA implementations, so you shouldn't need to be writing your own.
Presumably if you have two threads accessing one List (a Queue or ConcurrentQueue might have been better), you have some locking around it. Are you sure you're not holding a lock for a long time on one thread so the other thread gets blocked?
Ideally, you need to run a profiler of some kind, but you might be able to use Stopwatch and some trace to work out what's happening.
您可以在此处
尝试非加密哈希函数
它们应该比加密的更快
You could try out non-cryptographic hash functions here
They should be faster than the cryptographic ones
这是一个相对简单但有趣的性能谜题，不是吗？
在我看来，您在哈希算法实现中确实遇到了 CPU 瓶颈。对于高性能哈希算法，不要只是随机复制某些内容，而是使用 System.Security.Cryptography 中的标准类，例如
SHA1CryptoServiceProvider和MD5CryptoServiceProvider。如果您有多个可用核心，请考虑将哈希工作拆分为单独的线程。作为一般经验法则，对于
n个核心* 使用n+1线程；如果您有多线程核心（例如 Intel HT），则使用它们可能会提高或降低性能。 任务并行库可能对此提供帮助，特别是因为输入读取循环可以轻松重写为 迭代器。* 例如，在 Pentium IV Prescott 芯片上，在标准比特币客户端中使用两个内核会损失大约 10% 的性能，该客户端主要位于运行哈希的循环中。
This is a relatively simple yet interesting performance puzzle, isn't it?
It certainly sounds to me like you are CPU-bottlenecked in the hash algorithm implementation. For a performant hash algorithm, rather than just copying something random, use the standard classes in
System.Security.Cryptographysuch asSHA1CryptoServiceProviderandMD5CryptoServiceProvider.If you have multiple cores available, consider splitting up the hashing work into separate threads. As a general rule of thumb, for
ncores* usen+1threads; if you have multithreaded cores (such as Intel HT), you may gain or lose performance by using them. The Task Parallel Library may help with this, especially since the input read loop can be easily rewritten as an iterator.* For example, on a Pentium IV Prescott chip, approximately 10% performance is lost by using both cores in the standard Bitcoin client, which largely sits in a loop running hashes.