文件完整性检查
我们有大量（10+）非常大的文件（> 1 GB），我们通过网络共享。偶尔也会有一些较小的文件。
无论如何，我们希望确保这些文件在传输后正确无误。所以我想我可以创建一个工具来计算几个不同的哈希值（SHA1 和 MD5，也许还有 CRC32）并将其保存在元数据文件中。无论接收端是谁，都可以使用该工具验证文件。
我是否应该担心考虑到文件的大小，损坏的文件可能会与原始文件产生冲突？我不担心有人故意产生碰撞。
在这种情况下，使用 bittorrent 也是一种选择，因为它计算更小的块（几 KB 到几 MB）的 SHA1。另一个优点是它甚至可以处理数据传输。但它也增加了不便，因为相同的块可能属于不同的文件 - 在这种情况下，您需要文件的“邻居”来验证它。当然，我可以添加虚拟填充，但现在我让事情变得复杂了。
我知道 2^80 甚至 2^64 都是相当大的数字，并且哈希值被设计为具有“雪崩效应”，所以也许我只是有点偏执。
We have large sets (10+) of very large files (> 1 GB) that we share across our network. There are occasionally a few smaller files too.
Regardless, we want to make sure these files are correct after the transfers. So I am thinking that I could just create a tool that computes a couple of different hashes (SHA1 and MD5, maybe CRC32 too) and saves that in a metadata file. Whoever is on the receiving end can verify the files using the tool.
Should I be concerned that given the size of the files, it is possible that a broken file generates a collision with the original file? I am not worried about someone intentionally generating a collision.
In that case, using bittorrent is an option as well, since it calculates the SHA1 of much smaller chunks (a few KB to a few MB). Another plus is that it will even handle transmission of data. But it also adds an inconvenience, because the same chunk could belong to different files - in which case you need a file's "neighbors" to verify it. Of course I could add dummy padding, but now I am complicating things.
I know 2^80 and even 2^64 are pretty large numbers and that hashes are designed to have an "avalanche effect", so maybe I am just being a little paranoid.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（2）
最好的选择可能是使用专门为复制大文件而设计的实用程序，例如 Robocopy 或 Teracopy。
除了验证文件完整性之外，这些实用程序还消除了由于 Windows 资源管理器文件对话框意外弹出而导致副本中止的恼人情况。
Your best bet might be to use a utility specifically designed for copying large files, like Robocopy or Teracopy.
In addition to verifying file integrity, these utilities also eliminate the annoying cases where copies are aborted due to Windows Explorer file dialogs popping up unexpectedly.
新版本的 Ultracopier 也可以做到这一点。
演示视频显示断开连接并断开连接后恢复，文件完整性正确。
Ultracopier同样有很多数据保护机制。
The new version of Ultracopier can do that's too.
The video of presentation show disconnection and resume after disconnection, and the file integrity is correct.
Ultracopier same lot of data-protection mechanism.