java中最好的多部分base 64编码器是什么？
我测试了不同的base64编码器mig64、iHarder、sun等。似乎这些需要将整个数据存储在内存中进行转换。
如果我想编码一个大文件（流）> 1gb 以多线程方式，可以使用哪种编解码器实现而不损坏文件？ commons 编解码器似乎有 base64outputstream 包装器。还有其他解决方案吗？
说清楚一点，我有一个1TB的文件，这个文件需要进行base64编码。机器内存2GB RAM，用Java最快的方法是什么？
I have tested different base64 encoders mig64,iHarder,sun etc. Seems like these need to have the whole data need to be in memory for conversion.
If I want to encode a large file (stream) > 1gb in a multi-threaded fashion, which codec implementation can be used without corrupting the file? commons codec seems to have the base64outputstream wrapper. any other solutions?
To make it clear, I have a 1TB file, and this file need to be encoded base64. Machine memory 2GB RAM, what is the fastest way to do it in Java?
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（1）
我不确定哪个编码器更快，您必须测量每个编码器才能确定。但是，您可以通过将文件拆分为块来避免内存问题并实现并发。只需确保在某个 6 字节边界上分割它们（因为它均匀地变成 Base64 中的 8 字节）。
我建议选择合理的块大小并使用
ExecutorService管理固定数量的线程来做处理。您可以共享RandomAccessFile< /a> 在它们之间并写到适当的地方。当然，您必须计算输出块偏移量（只需乘以 8 再除以 6）。老实说，尽管您可能不会意识到并发性带来了多大的性能提升。它可能会因随机访问而压垮硬盘。我首先使用单个线程对文件进行分块。首先看看速度有多快。您处理 1GB 文件的速度可能比您想象的要快。作为粗略的猜测，我会说在现代硬件上需要 1 分钟，甚至写入您正在读取的同一驱动器。
I'm not sure which encoder is faster offhand, you'll have to measure each to determine that. However you can avoid the memory problem and accomplish the concurrency by splitting the file into chunks. Just make sure you split them on some 6-byte boundary (since it evenly turns into 8 bytes in Base64).
I'd recommend picking a reasonable chunk size and using an
ExecutorServiceto manage a fixed number of threads to do the processing. You can share aRandomAccessFilebetween them and write to the appropriate places. You'll of course have to calculate the output chunk offsets (just multiple by 8 and divide by 6).Honestly though you might not realize much performance gain here with concurrency. It could just overwhelm the hard drive with random access. I'd start with chunking the file up using a single thread. See how fast that is first. You can probably crunch a 1GB file faster than you think. As a rough guess I'd say 1 minute on modern hardware, even writing to the same drive you're reading from.