当前位置： 文江博客话题详情

如何将一个巨大的 zip 文件分割成多个卷？

发布于 2024-07-08 04:52:05 字数 211 浏览 7 评论 0原文

当我通过 java.util.zip.* 创建 zip 存档时，有没有办法将生成的存档拆分为多个卷？

假设我的整个存档的文件大小为24 MB，我想将其拆分为 3 个文件，每个文件限制为 10 MB。
有没有具有此功能的 zip API？ 或者还有其他好的方法来实现这一目标吗？

谢谢 托尔斯滕

原文

When I create a zip Archive via java.util.zip.*, is there a way to split the resulting archive in multiple volumes?

Let's say my overall archive has a filesize of 24 MB and I want to split it into 3 files on a limit of 10 MB per file.
Is there a zip API which has this feature? Or any other nice ways to achieve this?

Thanks
Thollsten

如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论， 你可以免费 注册 一个本站的账号。

评论（5）

如梦初醒的夏天 2024-07-15 04:52:05

检查： http:// saloon.javaranch.com/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic&f=38&t=004618

我不知道有任何公共 API 可以帮助您做到这一点。
（尽管如果您不想以编程方式执行此操作，则有像 WinSplitter 这样的实用程序可以执行此操作）

我还没有尝试过，但是，使用 ZippedInput/OutputStream 时的每个 ZipEntry 都有压缩的大小。 您可以在创建压缩文件时粗略估计其大小。 如果您需要 2MB 的压缩文件，那么您可以在条目的累积大小达到 1.9MB 后停止写入文件，为清单文件和其他 zip 文件特定元素占用 0.1MB。
简而言之，您可以在 ZippedInputStream 上编写一个包装器，如下所示：

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

public class ChunkedZippedOutputStream {

    private ZipOutputStream zipOutputStream;

    private final String path;
    private final String name;

    private long currentSize;
    private int currentChunkIndex;
    private final long MAX_FILE_SIZE = 16000000; // Whatever size you want
    private final String PART_POSTFIX = ".part.";
    private final String FILE_EXTENSION = ".zip";

    public ChunkedZippedOutputStream(String path, String name) throws FileNotFoundException {
        this.path = path;
        this.name = name;
        constructNewStream();
    }

    public void addEntry(ZipEntry entry) throws IOException {
        long entrySize = entry.getCompressedSize();
        if ((currentSize + entrySize) > MAX_FILE_SIZE) {
            closeStream();
            constructNewStream();
        } else {
            currentSize += entrySize;
            zipOutputStream.putNextEntry(entry);
        }
    }

    private void closeStream() throws IOException {
        zipOutputStream.close();
    }

    private void constructNewStream() throws FileNotFoundException {
        zipOutputStream = new ZipOutputStream(new FileOutputStream(new File(path, constructCurrentPartName())));
        currentChunkIndex++;
        currentSize = 0;
    }

    private String constructCurrentPartName() {
        // This will give names is the form of <file_name>.part.0.zip, <file_name>.part.1.zip, etc.
        return name + PART_POSTFIX + currentChunkIndex + FILE_EXTENSION;
    }
}

上面的程序只是方法的提示，无论如何都不是最终的解决方案。

Check: http://saloon.javaranch.com/cgi-bin/ubb/ultimatebb.cgi?ubb=get_topic&f=38&t=004618

I am not aware of any public API that will help you do that.
(Although if you do not want to do it programatically, there are utilities like WinSplitter that will do it)

I have not tried it but, every ZipEntry while using ZippedInput/OutputStream has a compressed size. You may get a rough estimate of the size of the zipped file while creating it. If you need 2MB of zipped files, then you can stop writing to a file after the cumulative size of entries become 1.9MB, taking .1MB for Manifest file and other zip file specific elements.
So, in a nutshell, you can write a wrapper over the ZippedInputStream as follows:

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.zip.ZipEntry;
import java.util.zip.ZipOutputStream;

public class ChunkedZippedOutputStream {

    private ZipOutputStream zipOutputStream;

    private final String path;
    private final String name;

    private long currentSize;
    private int currentChunkIndex;
    private final long MAX_FILE_SIZE = 16000000; // Whatever size you want
    private final String PART_POSTFIX = ".part.";
    private final String FILE_EXTENSION = ".zip";

    public ChunkedZippedOutputStream(String path, String name) throws FileNotFoundException {
        this.path = path;
        this.name = name;
        constructNewStream();
    }

    public void addEntry(ZipEntry entry) throws IOException {
        long entrySize = entry.getCompressedSize();
        if ((currentSize + entrySize) > MAX_FILE_SIZE) {
            closeStream();
            constructNewStream();
        } else {
            currentSize += entrySize;
            zipOutputStream.putNextEntry(entry);
        }
    }

    private void closeStream() throws IOException {
        zipOutputStream.close();
    }

    private void constructNewStream() throws FileNotFoundException {
        zipOutputStream = new ZipOutputStream(new FileOutputStream(new File(path, constructCurrentPartName())));
        currentChunkIndex++;
        currentSize = 0;
    }

    private String constructCurrentPartName() {
        // This will give names is the form of <file_name>.part.0.zip, <file_name>.part.1.zip, etc.
        return name + PART_POSTFIX + currentChunkIndex + FILE_EXTENSION;
    }
}

The above program is just a hint of the approach and not a final solution by any means.

回复 原文
迎风吟唱 2024-07-15 04:52:05

如果目标是让输出与 pkzip 和 winzip 兼容，我不知道有任何开源库可以做到这一点。 我们对我们的一个应用程序有类似的要求，我最终编写了我们自己的实现（与 zip 标准兼容）。 如果我记得的话，对我们来说最困难的事情是我们必须动态生成单独的文件（大多数 zip 实用程序的工作方式是创建大 zip 文件，然后返回并稍后分割它 - 这要容易得多花了大约一天的时间来编写，并花了两天的时间来调试。

zip 标准解释了文件格式的样子，如果您不害怕卷起袖子，那么这绝对是可行的。自己实现一个 zip 文件生成器，但您可以使用 Java 的 Deflator 类来生成压缩数据的段流。您必须自己生成文件和节标头，但它们只是字节 - 一旦您深入研究，这没什么难的。这

是 zip 规范 - K 部分包含您具体要查找的信息，但你还需要阅读 A、B、C 和 F 如果你正在处理非常大的文件（我们是），你还必须了解 Zip64 的内容 - 但对于 24 MB，你是。美好的。

如果您想深入尝试 - 如果您遇到问题，请回帖，我会看看是否可以提供一些指导。

If the goal is to have the output be compatible with pkzip and winzip, I'm not aware of any open source libraries that do this. We had a similar requirement for one of our apps, and I wound up writing our own implementation (compatible with the zip standard). If I recall, the hardest thing for us was that we had to generate the individual files on the fly (the way that most zip utilities work is they create the big zip file, then go back and split it later - that's a lot easier to implement. Took about a day to write and 2 days to debug.

The zip standard explains what the file format has to look like. If you aren't afraid of rolling up your sleeves a bit, this is definitely doable. You do have to implement a zip file generator yourself, but you can use Java's Deflator class to generate the segment streams for the compressed data. You'll have to generate the file and section headers yourself, but they are just bytes - nothing too hard once you dive in.

Here's the zip specification - section K has the info you are looking for specifically, but you'll need to read A, B, C and F as well. If you are dealing with really big files (We were), you'll have to get into the Zip64 stuff as well - but for 24 MB, you are fine.

If you want to dive in and try it - if you run into questions, post back and I'll see if I can provide some pointers.

回复 原文
挽手叙旧 2024-07-15 04:52:05

就其价值而言，我喜欢在任何地方使用try-with-resources。 如果您喜欢这种设计模式，那么您会喜欢这个。 此外，如果条目大于所需的部件尺寸，这还解决了空部件的问题。 在最坏的情况下，您至少将拥有与条目一样多的部分。

在：

my-archive.zip

输出：

my-archive.part1of3.zip
my-archive.part2of3.zip
my-archive.part3of3.zip

注意：我正在使用日志记录和 Apache Commons FilenameUtils，但请随意使用工具包中的内容。

/**
 * Utility class to split a zip archive into parts (not volumes)
 * by attempting to fit as many entries into a single part before
 * creating a new part. If a part would otherwise be empty because
 * the next entry won't fit, it will be added anyway to avoid empty parts.
 *
 * @author Eric Draken, 2019
 */
public class Zip
{
    private static final int DEFAULT_BUFFER_SIZE = 1024 * 4;

    private static final String ZIP_PART_FORMAT = "%s.part%dof%d.zip";

    private static final String EXT = "zip";

    private static final Logger logger = LoggerFactory.getLogger( MethodHandles.lookup().lookupClass() );

    /**
     * Split a large archive into smaller parts
     *
     * @param zipFile             Source zip file to split (must end with .zip)
     * @param outZipFile          Destination zip file base path. The "part" number will be added automatically
     * @param approxPartSizeBytes Approximate part size
     * @throws IOException Exceptions on file access
     */
    public static void splitZipArchive(
        @NotNull final File zipFile,
        @NotNull final File outZipFile,
        final long approxPartSizeBytes ) throws IOException
    {
        String basename = FilenameUtils.getBaseName( outZipFile.getName() );
        Path basePath = outZipFile.getParentFile() != null ? // Check if this file has a parent folder
            outZipFile.getParentFile().toPath() :
            Paths.get( "" );
        String extension = FilenameUtils.getExtension( zipFile.getName() );
        if ( !extension.equals( EXT ) )
        {
            throw new IllegalArgumentException( "The archive to split must end with ." + EXT );
        }

        // Get a list of entries in the archive
        try ( ZipFile zf = new ZipFile( zipFile ) )
        {
            // Silliness check
            long minRequiredSize = zipFile.length() / 100;
            if ( minRequiredSize > approxPartSizeBytes )
            {
                throw new IllegalArgumentException(
                    "Please select a minimum part size over " + minRequiredSize + " bytes, " +
                        "otherwise there will be over 100 parts."
                );
            }

            // Loop over all the entries in the large archive
            // to calculate the number of parts required
            Enumeration<? extends ZipEntry> enumeration = zf.entries();
            long partSize = 0;
            long totalParts = 1;
            while ( enumeration.hasMoreElements() )
            {
                long nextSize = enumeration.nextElement().getCompressedSize();
                if ( partSize + nextSize > approxPartSizeBytes )
                {
                    partSize = 0;
                    totalParts++;
                }
                partSize += nextSize;
            }

            // Silliness check: if there are more parts than there
            // are entries, then one entry will occupy one part by contract
            totalParts = Math.min( totalParts, zf.size() );

            logger.debug( "Split requires {} parts", totalParts );
            if ( totalParts == 1 )
            {
                // No splitting required. Copy file
                Path outFile = basePath.resolve(
                    String.format( ZIP_PART_FORMAT, basename, 1, 1 )
                );
                Files.copy( zipFile.toPath(), outFile );
                logger.debug( "Copied {} to {} (pass-though)", zipFile.toString(), outFile.toString() );
                return;
            }

            // Reset
            enumeration = zf.entries();

            // Split into parts
            int currPart = 1;
            ZipEntry overflowZipEntry = null;
            while ( overflowZipEntry != null || enumeration.hasMoreElements() )
            {
                Path outFilePart = basePath.resolve(
                    String.format( ZIP_PART_FORMAT, basename, currPart++, totalParts )
                );
                overflowZipEntry = writeEntriesToPart( overflowZipEntry, zf, outFilePart, enumeration, approxPartSizeBytes );
                logger.debug( "Wrote {}", outFilePart );
            }
        }
    }

    /**
     * Write an entry to the to the outFilePart
     *
     * @param overflowZipEntry    ZipEntry that didn't fit in the last part, or null
     * @param inZipFile           The large archive to split
     * @param outFilePart         The part of the archive currently being worked on
     * @param enumeration         Enumeration of ZipEntries
     * @param approxPartSizeBytes Approximate part size
     * @return Overflow ZipEntry, or null
     * @throws IOException File access exceptions
     */
    private static ZipEntry writeEntriesToPart(
        @Nullable ZipEntry overflowZipEntry,
        @NotNull final ZipFile inZipFile,
        @NotNull final Path outFilePart,
        @NotNull final Enumeration<? extends ZipEntry> enumeration,
        final long approxPartSizeBytes
    ) throws IOException
    {
        try (
            ZipOutputStream zos =
                new ZipOutputStream( new FileOutputStream( outFilePart.toFile(), false ) )
        )
        {
            long partSize = 0;
            byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
            while ( overflowZipEntry != null || enumeration.hasMoreElements() )
            {
                ZipEntry entry = overflowZipEntry != null ? overflowZipEntry : enumeration.nextElement();
                overflowZipEntry = null;

                long entrySize = entry.getCompressedSize();
                if ( partSize + entrySize > approxPartSizeBytes )
                {
                    if ( partSize != 0 )
                    {
                        return entry;    // Finished this part, but return the dangling ZipEntry
                    }
                    // Add the entry anyway if the part would otherwise be empty
                }
                partSize += entrySize;
                zos.putNextEntry( entry );

                // Get the input stream for this entry and copy the entry
                try ( InputStream is = inZipFile.getInputStream( entry ) )
                {
                    int bytesRead;
                    while ( (bytesRead = is.read( buffer )) != -1 )
                    {
                        zos.write( buffer, 0, bytesRead );
                    }
                }
            }
            return null;    // Finished splitting
        }
    }

For what it's worth, I like to use try-with-resources everywhere. If you are into that design pattern, then you will like this. Also, this solves the problem of empty parts if the entries are larger than the desired part size. You will at least have as many parts as entries in the worst case.

In:

my-archive.zip

Out:

my-archive.part1of3.zip
my-archive.part2of3.zip
my-archive.part3of3.zip

Note: I'm using logging and Apache Commons FilenameUtils, but feel free to use what you have in your toolkit.

/**
 * Utility class to split a zip archive into parts (not volumes)
 * by attempting to fit as many entries into a single part before
 * creating a new part. If a part would otherwise be empty because
 * the next entry won't fit, it will be added anyway to avoid empty parts.
 *
 * @author Eric Draken, 2019
 */
public class Zip
{
    private static final int DEFAULT_BUFFER_SIZE = 1024 * 4;

    private static final String ZIP_PART_FORMAT = "%s.part%dof%d.zip";

    private static final String EXT = "zip";

    private static final Logger logger = LoggerFactory.getLogger( MethodHandles.lookup().lookupClass() );

    /**
     * Split a large archive into smaller parts
     *
     * @param zipFile             Source zip file to split (must end with .zip)
     * @param outZipFile          Destination zip file base path. The "part" number will be added automatically
     * @param approxPartSizeBytes Approximate part size
     * @throws IOException Exceptions on file access
     */
    public static void splitZipArchive(
        @NotNull final File zipFile,
        @NotNull final File outZipFile,
        final long approxPartSizeBytes ) throws IOException
    {
        String basename = FilenameUtils.getBaseName( outZipFile.getName() );
        Path basePath = outZipFile.getParentFile() != null ? // Check if this file has a parent folder
            outZipFile.getParentFile().toPath() :
            Paths.get( "" );
        String extension = FilenameUtils.getExtension( zipFile.getName() );
        if ( !extension.equals( EXT ) )
        {
            throw new IllegalArgumentException( "The archive to split must end with ." + EXT );
        }

        // Get a list of entries in the archive
        try ( ZipFile zf = new ZipFile( zipFile ) )
        {
            // Silliness check
            long minRequiredSize = zipFile.length() / 100;
            if ( minRequiredSize > approxPartSizeBytes )
            {
                throw new IllegalArgumentException(
                    "Please select a minimum part size over " + minRequiredSize + " bytes, " +
                        "otherwise there will be over 100 parts."
                );
            }

            // Loop over all the entries in the large archive
            // to calculate the number of parts required
            Enumeration<? extends ZipEntry> enumeration = zf.entries();
            long partSize = 0;
            long totalParts = 1;
            while ( enumeration.hasMoreElements() )
            {
                long nextSize = enumeration.nextElement().getCompressedSize();
                if ( partSize + nextSize > approxPartSizeBytes )
                {
                    partSize = 0;
                    totalParts++;
                }
                partSize += nextSize;
            }

            // Silliness check: if there are more parts than there
            // are entries, then one entry will occupy one part by contract
            totalParts = Math.min( totalParts, zf.size() );

            logger.debug( "Split requires {} parts", totalParts );
            if ( totalParts == 1 )
            {
                // No splitting required. Copy file
                Path outFile = basePath.resolve(
                    String.format( ZIP_PART_FORMAT, basename, 1, 1 )
                );
                Files.copy( zipFile.toPath(), outFile );
                logger.debug( "Copied {} to {} (pass-though)", zipFile.toString(), outFile.toString() );
                return;
            }

            // Reset
            enumeration = zf.entries();

            // Split into parts
            int currPart = 1;
            ZipEntry overflowZipEntry = null;
            while ( overflowZipEntry != null || enumeration.hasMoreElements() )
            {
                Path outFilePart = basePath.resolve(
                    String.format( ZIP_PART_FORMAT, basename, currPart++, totalParts )
                );
                overflowZipEntry = writeEntriesToPart( overflowZipEntry, zf, outFilePart, enumeration, approxPartSizeBytes );
                logger.debug( "Wrote {}", outFilePart );
            }
        }
    }

    /**
     * Write an entry to the to the outFilePart
     *
     * @param overflowZipEntry    ZipEntry that didn't fit in the last part, or null
     * @param inZipFile           The large archive to split
     * @param outFilePart         The part of the archive currently being worked on
     * @param enumeration         Enumeration of ZipEntries
     * @param approxPartSizeBytes Approximate part size
     * @return Overflow ZipEntry, or null
     * @throws IOException File access exceptions
     */
    private static ZipEntry writeEntriesToPart(
        @Nullable ZipEntry overflowZipEntry,
        @NotNull final ZipFile inZipFile,
        @NotNull final Path outFilePart,
        @NotNull final Enumeration<? extends ZipEntry> enumeration,
        final long approxPartSizeBytes
    ) throws IOException
    {
        try (
            ZipOutputStream zos =
                new ZipOutputStream( new FileOutputStream( outFilePart.toFile(), false ) )
        )
        {
            long partSize = 0;
            byte[] buffer = new byte[DEFAULT_BUFFER_SIZE];
            while ( overflowZipEntry != null || enumeration.hasMoreElements() )
            {
                ZipEntry entry = overflowZipEntry != null ? overflowZipEntry : enumeration.nextElement();
                overflowZipEntry = null;

                long entrySize = entry.getCompressedSize();
                if ( partSize + entrySize > approxPartSizeBytes )
                {
                    if ( partSize != 0 )
                    {
                        return entry;    // Finished this part, but return the dangling ZipEntry
                    }
                    // Add the entry anyway if the part would otherwise be empty
                }
                partSize += entrySize;
                zos.putNextEntry( entry );

                // Get the input stream for this entry and copy the entry
                try ( InputStream is = inZipFile.getInputStream( entry ) )
                {
                    int bytesRead;
                    while ( (bytesRead = is.read( buffer )) != -1 )
                    {
                        zos.write( buffer, 0, bytesRead );
                    }
                }
            }
            return null;    // Finished splitting
        }
    }
回复 原文
兮子 2024-07-15 04:52:05

下面的代码是我的解决方案，根据所需的大小将目录结构中的 zip 文件拆分为块。 我发现以前的答案很有用，所以想用类似但更简洁的方法做出贡献。 该代码适合我的特定需求，并且我相信还有改进的空间。

import java.io.*;
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;

class ChunkedZip {
    private final static long MAX_FILE_SIZE = 1000 * 1000 * 1024; //  around 1GB 
    private final static String zipCopyDest = "C:\\zip2split\\copy";

    public static void splitZip(String zipFileName, String zippedPath, String coreId) throws IOException {

        System.out.println("process whole zip file..");
        FileInputStream fis = new FileInputStream(zippedPath);
        ZipInputStream zipInputStream = new ZipInputStream(fis);
        ZipEntry entry = null;
        int currentChunkIndex = 0;
        //using just to get the uncompressed size of the zipentries
        long entrySize = 0;
        ZipFile zipFile = new ZipFile(zippedPath);
        Enumeration enumeration = zipFile.entries();

        String copDest = zipCopyDest + "\\" + coreId + "_" + currentChunkIndex + ".zip";

        FileOutputStream fos = new FileOutputStream(new File(copDest));
        BufferedOutputStream bos = new BufferedOutputStream(fos);
        ZipOutputStream zos = new ZipOutputStream(bos);
        long currentSize = 0;

        try {
            while ((entry = zipInputStream.getNextEntry()) != null && enumeration.hasMoreElements()) {

                ZipEntry zipEntry = (ZipEntry) enumeration.nextElement();
                System.out.println(zipEntry.getName());
                System.out.println(zipEntry.getSize());
                entrySize = zipEntry.getSize();

                ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
                //long entrySize = entry.getCompressedSize();
                //entrySize = entry.getSize(); //gives -1

                if ((currentSize + entrySize) > MAX_FILE_SIZE) {
                    zos.close();
                    //construct a new stream
                    //zos = new ZipOutputStream(new FileOutputStream(new File(zippedPath, constructCurrentPartName(coreId))));
                    currentChunkIndex++;
                    zos = getOutputStream(currentChunkIndex, coreId);
                    currentSize = 0;

                } else {
                    currentSize += entrySize;
                    zos.putNextEntry(new ZipEntry(entry.getName()));
                    byte[] buffer = new byte[8192];
                    int length = 0;
                    while ((length = zipInputStream.read(buffer)) > 0) {
                        outputStream.write(buffer, 0, length);
                    }

                    byte[] unzippedFile = outputStream.toByteArray();
                    zos.write(unzippedFile);
                    unzippedFile = null;
                    outputStream.close();
                    zos.closeEntry();
                }
                //zos.close();
            }
        } finally {
            zos.close();
        }
    }

    public static ZipOutputStream getOutputStream(int i, String coreId) throws IOException {
        System.out.println("inside of getOutputStream()..");
        ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipCopyDest + "\\" + coreId + "_" + i + ".zip"));
        // out.setLevel(Deflater.DEFAULT_COMPRESSION);
        return out;
    }

    public static void main(String args[]) throws IOException {
        String zipFileName = "Large_files_for_testing.zip";
        String zippedPath = "C:\\zip2split\\Large_files_for_testing.zip";
        String coreId = "Large_files_for_testing";
        splitZip(zipFileName, zippedPath, coreId);
    }
}

Below code is my solution to split zip file in directory structure to chunks based on desired size. I found the previous answers useful so, wanted to contribute with similar but little more neat approach. This code is working for me for my specific needs, and I believe there is room for improvement.

import java.io.*;
import java.util.Enumeration;
import java.util.zip.ZipEntry;
import java.util.zip.ZipFile;
import java.util.zip.ZipInputStream;
import java.util.zip.ZipOutputStream;

class ChunkedZip {
    private final static long MAX_FILE_SIZE = 1000 * 1000 * 1024; //  around 1GB 
    private final static String zipCopyDest = "C:\\zip2split\\copy";

    public static void splitZip(String zipFileName, String zippedPath, String coreId) throws IOException {

        System.out.println("process whole zip file..");
        FileInputStream fis = new FileInputStream(zippedPath);
        ZipInputStream zipInputStream = new ZipInputStream(fis);
        ZipEntry entry = null;
        int currentChunkIndex = 0;
        //using just to get the uncompressed size of the zipentries
        long entrySize = 0;
        ZipFile zipFile = new ZipFile(zippedPath);
        Enumeration enumeration = zipFile.entries();

        String copDest = zipCopyDest + "\\" + coreId + "_" + currentChunkIndex + ".zip";

        FileOutputStream fos = new FileOutputStream(new File(copDest));
        BufferedOutputStream bos = new BufferedOutputStream(fos);
        ZipOutputStream zos = new ZipOutputStream(bos);
        long currentSize = 0;

        try {
            while ((entry = zipInputStream.getNextEntry()) != null && enumeration.hasMoreElements()) {

                ZipEntry zipEntry = (ZipEntry) enumeration.nextElement();
                System.out.println(zipEntry.getName());
                System.out.println(zipEntry.getSize());
                entrySize = zipEntry.getSize();

                ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
                //long entrySize = entry.getCompressedSize();
                //entrySize = entry.getSize(); //gives -1

                if ((currentSize + entrySize) > MAX_FILE_SIZE) {
                    zos.close();
                    //construct a new stream
                    //zos = new ZipOutputStream(new FileOutputStream(new File(zippedPath, constructCurrentPartName(coreId))));
                    currentChunkIndex++;
                    zos = getOutputStream(currentChunkIndex, coreId);
                    currentSize = 0;

                } else {
                    currentSize += entrySize;
                    zos.putNextEntry(new ZipEntry(entry.getName()));
                    byte[] buffer = new byte[8192];
                    int length = 0;
                    while ((length = zipInputStream.read(buffer)) > 0) {
                        outputStream.write(buffer, 0, length);
                    }

                    byte[] unzippedFile = outputStream.toByteArray();
                    zos.write(unzippedFile);
                    unzippedFile = null;
                    outputStream.close();
                    zos.closeEntry();
                }
                //zos.close();
            }
        } finally {
            zos.close();
        }
    }

    public static ZipOutputStream getOutputStream(int i, String coreId) throws IOException {
        System.out.println("inside of getOutputStream()..");
        ZipOutputStream out = new ZipOutputStream(new FileOutputStream(zipCopyDest + "\\" + coreId + "_" + i + ".zip"));
        // out.setLevel(Deflater.DEFAULT_COMPRESSION);
        return out;
    }

    public static void main(String args[]) throws IOException {
        String zipFileName = "Large_files_for_testing.zip";
        String zippedPath = "C:\\zip2split\\Large_files_for_testing.zip";
        String coreId = "Large_files_for_testing";
        splitZip(zipFileName, zippedPath, coreId);
    }
}
回复 原文
静谧幽蓝 2024-07-15 04:52:05

这是我的解决方案：

public abstract class ZipHelper {

    public static NumberFormat formater = NumberFormat.getNumberInstance(new Locale("pt", "BR"));

    public static List<Path> zip(Collection<File> inputFiles, long maxSize) throws IOException {

        byte[] buffer = new byte[1024];
        int count = 0;
        long currentZipSize = maxSize;
        List<Path> response = new ArrayList<>();
        ZipOutputStream zip = null;
        for (File currentFile : inputFiles) {
            long nextFileSize = currentFile.length();
            long predictedZipSize = currentZipSize + nextFileSize;
            boolean needNewFile = predictedZipSize >= maxSize;
            System.out.println("[=] ZIP current (" + formater.format(currentZipSize) + ") + next file (" + formater.format(nextFileSize) + ") = predicted (" + formater.format(predictedZipSize) + ") > max (" + formater.format(maxSize) + ") ? " + needNewFile);
            if (needNewFile) {
                safeClose(zip);
                Path tmpFile = Files.createTempFile("teste-", (".part." + count++ + ".zip"));
                System.out.println("[#] Starting new file: " + tmpFile);
                zip = new ZipOutputStream(Files.newOutputStream(tmpFile));
                zip.setLevel(Deflater.BEST_COMPRESSION);
                response.add(tmpFile);
                currentZipSize = 0;
            }
            ZipEntry zipEntry = new ZipEntry(currentFile.getName());
            System.out.println("[<] Adding to ZIP: " + currentFile.getName());
            zip.putNextEntry(zipEntry);
            FileInputStream in = new FileInputStream(currentFile);
            zip.write(in.readAllBytes());
            zip.closeEntry();
            safeClose(in);
            long compressed = zipEntry.getCompressedSize();
            System.out.println("[=] Compressed current file: " + formater.format(compressed));
            currentZipSize += zipEntry.getCompressedSize();
        }
        safeClose(zip);
        return response;
    }

    public static void safeClose(Closeable... closeables) {
        if (closeables != null) {
            for (Closeable closeable : closeables) {
                if (closeable != null) {
                    try {
                        System.out.println("[X] Closing: (" + closeable.getClass() + ") - " + closeable);
                        closeable.close();
                    } catch (Throwable ex) {
                        System.err.println("[!] Error on close: " + closeable);
                        ex.printStackTrace();
                    }
                }
            }
        }
    }
}

控制台输出：

[?] Files to process: [\data\teste\TestFile(1).pdf, \data\teste\TestFile(2).pdf, \data\teste\TestFile(3).pdf, \data\teste\TestFile(4).pdf, \data\teste\TestFile(5).pdf, \data\teste\TestFile(6).pdf, \data\teste\TestFile(7).pdf]
[=] ZIP current (3.145.728) + next file (1.014.332) = predicted (4.160.060) > max (3.145.728) ? true
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-3319961516431535912.part.0.zip
[<] Adding to ZIP: TestFile(1).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@3d99d22e
[=] Compressed current file: 940.422
[=] ZIP current (940.422) + next file (1.511.862) = predicted (2.452.284) > max (3.145.728) ? false
[<] Adding to ZIP: TestFile(2).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@49fc609f
[=] Compressed current file: 1.475.178
[=] ZIP current (2.415.600) + next file (2.439.287) = predicted (4.854.887) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@cd2dae5
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-8849887746791381380.part.1.zip
[<] Adding to ZIP: TestFile(3).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@4973813a
[=] Compressed current file: 2.374.718
[=] ZIP current (2.374.718) + next file (2.385.447) = predicted (4.760.165) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@6321e813
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-6305809161676875106.part.2.zip
[<] Adding to ZIP: TestFile(4).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@79be0360
[=] Compressed current file: 2.202.203
[=] ZIP current (2.202.203) + next file (292.918) = predicted (2.495.121) > max (3.145.728) ? false
[<] Adding to ZIP: TestFile(5).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@22a67b4
[=] Compressed current file: 230.491
[=] ZIP current (2.432.694) + next file (4.197.512) = predicted (6.630.206) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@57855c9a
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-17160527941340008316.part.3.zip
[<] Adding to ZIP: TestFile(6).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@3b084709
[=] Compressed current file: 3.020.115
[=] ZIP current (3.020.115) + next file (1.556.237) = predicted (4.576.352) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@3224f60b
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-14050058835776413808.part.4.zip
[<] Adding to ZIP: TestFile(7).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@63e2203c
[=] Compressed current file: 1.460.566
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@1efed156
[>] Generated ZIP files(s): [C:\Users\Cassio\AppData\Local\Temp\teste-3319961516431535912.part.0.zip, C:\Users\Cassio\AppData\Local\Temp\teste-8849887746791381380.part.1.zip, C:\Users\Cassio\AppData\Local\Temp\teste-6305809161676875106.part.2.zip, C:\Users\Cassio\AppData\Local\Temp\teste-17160527941340008316.part.3.zip, C:\Users\Cassio\AppData\Local\Temp\teste-14050058835776413808.part.4.zip]

Here's my solution:

public abstract class ZipHelper {

    public static NumberFormat formater = NumberFormat.getNumberInstance(new Locale("pt", "BR"));

    public static List<Path> zip(Collection<File> inputFiles, long maxSize) throws IOException {

        byte[] buffer = new byte[1024];
        int count = 0;
        long currentZipSize = maxSize;
        List<Path> response = new ArrayList<>();
        ZipOutputStream zip = null;
        for (File currentFile : inputFiles) {
            long nextFileSize = currentFile.length();
            long predictedZipSize = currentZipSize + nextFileSize;
            boolean needNewFile = predictedZipSize >= maxSize;
            System.out.println("[=] ZIP current (" + formater.format(currentZipSize) + ") + next file (" + formater.format(nextFileSize) + ") = predicted (" + formater.format(predictedZipSize) + ") > max (" + formater.format(maxSize) + ") ? " + needNewFile);
            if (needNewFile) {
                safeClose(zip);
                Path tmpFile = Files.createTempFile("teste-", (".part." + count++ + ".zip"));
                System.out.println("[#] Starting new file: " + tmpFile);
                zip = new ZipOutputStream(Files.newOutputStream(tmpFile));
                zip.setLevel(Deflater.BEST_COMPRESSION);
                response.add(tmpFile);
                currentZipSize = 0;
            }
            ZipEntry zipEntry = new ZipEntry(currentFile.getName());
            System.out.println("[<] Adding to ZIP: " + currentFile.getName());
            zip.putNextEntry(zipEntry);
            FileInputStream in = new FileInputStream(currentFile);
            zip.write(in.readAllBytes());
            zip.closeEntry();
            safeClose(in);
            long compressed = zipEntry.getCompressedSize();
            System.out.println("[=] Compressed current file: " + formater.format(compressed));
            currentZipSize += zipEntry.getCompressedSize();
        }
        safeClose(zip);
        return response;
    }

    public static void safeClose(Closeable... closeables) {
        if (closeables != null) {
            for (Closeable closeable : closeables) {
                if (closeable != null) {
                    try {
                        System.out.println("[X] Closing: (" + closeable.getClass() + ") - " + closeable);
                        closeable.close();
                    } catch (Throwable ex) {
                        System.err.println("[!] Error on close: " + closeable);
                        ex.printStackTrace();
                    }
                }
            }
        }
    }
}

And the console output:

[?] Files to process: [\data\teste\TestFile(1).pdf, \data\teste\TestFile(2).pdf, \data\teste\TestFile(3).pdf, \data\teste\TestFile(4).pdf, \data\teste\TestFile(5).pdf, \data\teste\TestFile(6).pdf, \data\teste\TestFile(7).pdf]
[=] ZIP current (3.145.728) + next file (1.014.332) = predicted (4.160.060) > max (3.145.728) ? true
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-3319961516431535912.part.0.zip
[<] Adding to ZIP: TestFile(1).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@3d99d22e
[=] Compressed current file: 940.422
[=] ZIP current (940.422) + next file (1.511.862) = predicted (2.452.284) > max (3.145.728) ? false
[<] Adding to ZIP: TestFile(2).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@49fc609f
[=] Compressed current file: 1.475.178
[=] ZIP current (2.415.600) + next file (2.439.287) = predicted (4.854.887) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@cd2dae5
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-8849887746791381380.part.1.zip
[<] Adding to ZIP: TestFile(3).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@4973813a
[=] Compressed current file: 2.374.718
[=] ZIP current (2.374.718) + next file (2.385.447) = predicted (4.760.165) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@6321e813
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-6305809161676875106.part.2.zip
[<] Adding to ZIP: TestFile(4).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@79be0360
[=] Compressed current file: 2.202.203
[=] ZIP current (2.202.203) + next file (292.918) = predicted (2.495.121) > max (3.145.728) ? false
[<] Adding to ZIP: TestFile(5).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@22a67b4
[=] Compressed current file: 230.491
[=] ZIP current (2.432.694) + next file (4.197.512) = predicted (6.630.206) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@57855c9a
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-17160527941340008316.part.3.zip
[<] Adding to ZIP: TestFile(6).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@3b084709
[=] Compressed current file: 3.020.115
[=] ZIP current (3.020.115) + next file (1.556.237) = predicted (4.576.352) > max (3.145.728) ? true
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@3224f60b
[#] Starting new file: C:\Users\Cassio\AppData\Local\Temp\teste-14050058835776413808.part.4.zip
[<] Adding to ZIP: TestFile(7).pdf
[X] Closing: (class java.io.FileInputStream) - java.io.FileInputStream@63e2203c
[=] Compressed current file: 1.460.566
[X] Closing: (class java.util.zip.ZipOutputStream) - java.util.zip.ZipOutputStream@1efed156
[>] Generated ZIP files(s): [C:\Users\Cassio\AppData\Local\Temp\teste-3319961516431535912.part.0.zip, C:\Users\Cassio\AppData\Local\Temp\teste-8849887746791381380.part.1.zip, C:\Users\Cassio\AppData\Local\Temp\teste-6305809161676875106.part.2.zip, C:\Users\Cassio\AppData\Local\Temp\teste-17160527941340008316.part.3.zip, C:\Users\Cassio\AppData\Local\Temp\teste-14050058835776413808.part.4.zip]
回复 原文
~没有更多了~

关于作者

匿名。

暂无简介

文章
评论
26 人气
发私信

相关话题

更多

热门标签

操作系统 程序设计 IT运维 Linux系统管理 JavaScript 服务器应用 solaris C/C++ PHP Shell BSD Vue.js aix Oracle Python HTML 系统管理 HTML5 CSS 前端
更多

推荐作者

lee_heart

文章 0 评论 0

不喜欢何必死缠烂打

文章 0 评论 0

huangxaiorui

文章 0 评论 0

ゞ记忆︶ㄣ

文章 0 评论 0

画离情绘悲伤

文章 0 评论 0

渚

文章 0 评论 0

更多

友情链接

文江博客
    我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站，即表示您同意使用 Cookies 和您的相关数据。
    原文