记录到 Amazon S3
有没有人构建过或者知道如何构建将日志文件存储在 Amazon S3 上的日志记录框架？
我们正在构建一个 SaaS 应用程序，自然地，我们计划拥有无数的服务器和客户。我想简化整个日志记录结构。目前我们使用 SLF4j 进行日志记录，并使用 Logback 作为实际的日志记录实现。
我想我们可以引入另一个实现，该实现会在内存中累积日志消息，压缩它们，然后将它们上传到 S3。如果 S3 连接中断或速度缓慢，实施会将文件在磁盘上排队。有点像 Facebook 的 Scribe。
我的猜测是我需要四五天的时间来编写和测试它。如果还有其他实现，我很想知道。
Has anyone built, or does anyone know how to build, a logging framework that deposits the log files on Amazon S3?
We're building a SaaS app, and, naturally, we plan to have a zillion servers and customers. I'd like to simplify the entire logging structure. At the moment we use SLF4j for logging and Logback as the actual logging implementation.
I'm thinking that we could drop in another implementation that would accumulate log messages in memory, compress them, and then upload them to S3. If the S3 connection were down or slow, the implementation would queue the files up on disk. Kind of like Facebook's Scribe.
My guess is that it would take me four or five days to write and test this. If there's another implementation out there, I'd love to know about it.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（5）
fluidd 有一个插件可以将文件存储到 s3。
（Fluentd 是一个很好的“日志文件收集器”）
在这里阅读更多相关信息：
https://docs.fluenced.org/output/s3
如果 s3 连接已关闭或慢它会缓冲输出
为你。
There is a plugin for fluentd that stores files to s3.
(Fluentd is a nice "log file collector")
Read more about it here:
https://docs.fluentd.org/output/s3
If the s3 connection is down or slow it will buffer the output
for you.
您可以尝试为
logback或log4j2编写自定义appender并在相应的配置中使用此 Appender。这样，您不必编写整个日志记录框架，而只需编写您需要的部分，并使用工作框架中的其余部分。
github 上也有一些。例如： shuwada/logback-s3
You can try to write a custom
appenderforlogbackorlog4j2and use this appender in respective configuration.This way, you don't have to write an entire logging framework, but only the part you need and use the rest from a working framework.
There are also a few of them on github. For instance: shuwada/logback-s3
我在谷歌上搜索同样的问题。但显然我的情况稍微好一些。
我知道如何登录到 S3。没有开箱即用的解决方案。
我建议s3使用类似fuse的东西来挂载你的系统日志： https://github.com/s3fs- fusion/s3fs-fuse
现在，您所需要的就是将其也安装到日志解析器系统中，该系统可以是任何可以从目录读取日志的现成系统。
这就是我想出来的。
在实施之前我仍在寻找的是这种记录到 S3 的性能问题，因为 AWS 有它自己的问题。讨论了设置块/文件大小以提高性能并降低读/写成本。
希望它能帮助另一个迷失的灵魂。
I was searching google for the same question. But apparently I am slightly better off.
I know how to log to S3. There is no out-of-the-box solution.
I suggest something like fuse for s3 to mount your syslog: https://github.com/s3fs-fuse/s3fs-fuse
Now all you need is to mount it also in your log-parser system which can be any of the off-the-shelf system that can read logs off of a directory.
This is what I came up with.
What I am still searching for before implementing is the performance issue of such logging to S3 since AWS has it's own issues. There was a discussion on setting block/file size to improve performance and lowering read/write cost.
Hope it helps another lost soul.
我一直在寻找类似的东西。
我已经使用 winston.js 进行日志记录，但我发现这个插件可以让您将日志保存到 AWS S3。 (Winston-S3)
我还没有尝试过，但我很快就会尝试。
在某些管理页面中显示这些日志应该不难。
I was looking for something similar.
I'm using winston.js for logging purposes already but I found this plugin that let's you save your logs to AWS S3. (Winston-S3)
I haven't tried it yet but I will do shortly.
It shouldn't be to difficult to show those logs in some admin page.
您可以登录Datadog，然后配置日志归档。例如，您可以让容器将日志以 JSON 的形式输出到 stdout，然后让 Datadog 代理或 Fluentd/Fluentbit 将这些日志转发到 Datadog。 Datadog会自动解析并索引日志以进行快速搜索。如果您编写自己的解析规则，不同的日志记录格式也可以工作。在保留期结束时，如果您设置了日志归档，它会自动将它们上传到 S3。
缺点是受到 Datadog 和价格的限制。优点是您可以轻松地将日志重新合并回 Datadog 中以进行快速搜索，并且您不必担心维护自托管解决方案。
如果您想要一个开源替代方案，您可以尝试 Loki。它有一个S3存储后端。
You could log to Datadog and then configure log archiving. For example, you could have containers output logs to stdout as JSON and then have the Datadog agent or fluentd/fluentbit forward logs those logs to Datadog. Datadog would automatically parse and index the logs for rapid searching. Different logging formats also work if you write your own parsing rules. At the end of retention period if you have log archiving set it will automatically upload them to S3 for you.
The disadvantage is being locked into Datadog and the price. The advantage is you can easily re-hydrate the logs back into Datadog for fast searching and you don't have to worry about maintaing a self-hosted solution.
If you want an open-source alternative you could try out Loki. It has a S3 storage backend.