使用协议缓冲区进行二进制日志记录

发布于 2024-08-25 00:35:09 字数 715 浏览 4 评论 0原文

我们正在考虑使用协议缓冲区进行二进制日志记录，因为：

这就是我们编码对象的方式。
它相对紧凑，读/写速度快等。

也就是说，我们应该如何处理它并不明显，因为API 往往专注于创建整个对象，因此将 DataLogEntry 列表包装为 DataLogFile 中的重复字段将是您在消息传递术语中要做的事情，但我们真正想要的只是能够写入然后读取整个 DataLogEntry out，将其附加到文件末尾。

我们这样做遇到的第一个问题是这样做（在测试中：

        FileInputStream fileIn = new FileInputStream(logFile);
        CodedInputStream in = CodedInputStream.newInstance(fileIn);
        while(!in.isAtEnd()) {
            DataLogEntry entry = DataLogEntry.parseFrom(in);
            // ... do stuff
        }

仅导致从流中读取 1 个 DataLogEntry。如果没有 isAtEnd，它永远不会停止。

想法？

编辑：我已切换到使用条目。 writeDelimitedTo 和 BidLogEntry.parseDelimitedFrom 这似乎有效......

原文

We're thinking of using Protocol Buffers for binary logging because:

It's how we're encoding our objects anyway
It is relatively compact, fast to read / write etc.

That said, it isn't obvious how we should go about it because the APIs tend to focus on creating whole objects, so wrapping a list of DataLogEntry as a repeated field in a DataLogFile would be what you'd do in messaging terms but what we really want is just to be able to write and then read a whole DataLogEntry out, appending it to the end of a file.

The first issue we've hit by doing that is that doing this (in a test:

        FileInputStream fileIn = new FileInputStream(logFile);
        CodedInputStream in = CodedInputStream.newInstance(fileIn);
        while(!in.isAtEnd()) {
            DataLogEntry entry = DataLogEntry.parseFrom(in);
            // ... do stuff
        }

Only results in 1 DataLogEntry being read from the stream. Without the isAtEnd, it never stops.

Thoughts?

Edit: I've switched to using entry.writeDelimitedTo and BidLogEntry.parseDelimitedFrom and that seems to work...

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

素食主义者 2024-09-01 00:35:09

根据我对协议缓冲区的理解，它不支持单流。因此，您可能需要自己跟踪消息的边界。您可以通过在日志中每条消息之前存储消息的大小来实现此目的。

public class DataLog {

    public void write(final DataOutputStream out, final DataLogEntry entry) throws IOException {
        out.writeInt(entry.getSerializedSize());
        CodedOutputStream codedOut = CodedOutputStream.newInstance(out);
        entry.writeTo(codedOut);
        codedOut.flush();
    }

    public void read(final DataInputStream in) throws IOException {
        byte[] buffer = new byte[4096];
        while (true) {
            try {
                int size = in.readInt();
                CodedInputStream codedIn;
                if (size <= buffer.length) {
                    in.read(buffer, 0, size);
                    codedIn = CodedInputStream.newInstance(buffer, 0, size);
                } else {
                    byte[] tmp = new byte[size];
                    in.read(tmp);
                    codedIn = CodedInputStream.newInstance(tmp);
                }
                DataLogEntry.parseFrom(codedIn);
                // ... do stuff
            }
            catch (final EOFException e) {
                break;
            }
        }
    }
}

注意：我使用 EOFException 来查找文件末尾，您可能希望使用分隔符或手动跟踪读取的字节数。

From my understanding of protocol buffers it does not support multiple messages in a single stream. So you will probably need to track the boundaries of the messages yourself. you can do this by storing the size of the message before each message in the log.

public class DataLog {

    public void write(final DataOutputStream out, final DataLogEntry entry) throws IOException {
        out.writeInt(entry.getSerializedSize());
        CodedOutputStream codedOut = CodedOutputStream.newInstance(out);
        entry.writeTo(codedOut);
        codedOut.flush();
    }

    public void read(final DataInputStream in) throws IOException {
        byte[] buffer = new byte[4096];
        while (true) {
            try {
                int size = in.readInt();
                CodedInputStream codedIn;
                if (size <= buffer.length) {
                    in.read(buffer, 0, size);
                    codedIn = CodedInputStream.newInstance(buffer, 0, size);
                } else {
                    byte[] tmp = new byte[size];
                    in.read(tmp);
                    codedIn = CodedInputStream.newInstance(tmp);
                }
                DataLogEntry.parseFrom(codedIn);
                // ... do stuff
            }
            catch (final EOFException e) {
                break;
            }
        }
    }
}

NB: I've used an EOFException to find the end of file, you may wish to use a delimiter or track the number of byte read manually.

回复收藏 0 原文