从远程计算机收集日志文件的最佳方法?

发布于 2024-07-12 08:54:27 字数 272 浏览 9 评论 0原文

我有 500 多台计算机分布在覆盖三大洲的 WAN 上。 我需要定期收集每个刀片本地硬盘上的文本文件。 每台服务器都运行 Windows Server 2003,文件安装在可作为 \server\Logs 远程访问的共享上。 每台机器保存许多文件,每个文件可能有几Mb,并且可以通过压缩来减小大小。

到目前为止,我已经尝试使用 Powershell 脚本和简单的 Java 应用程序来进行复制。 两种方法都需要几天时间才能收集 500Gb 左右的文件。 有没有更好、更快、更高效的解决方案?

I have over 500 machines distributed across a WAN covering three continents. Periodically, I need to collect text files which are on the local hard disk on each blade. Each server is running Windows server 2003 and the files are mounted on a share which can be accessed remotely as \server\Logs. Each machine holds many files which can be several Mb each and the size can be reduced by zipping.

Thus far I have tried using Powershell scripts and a simple Java application to do the copying. Both approaches take several days to collect the 500Gb or so of files. Is there a better solution which would be faster and more efficient?

如果你对这篇内容有疑问,欢迎到本站社区发帖提问 参与讨论,获取更多帮助,或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论, 你可以免费 注册 一个本站的账号。

评论(7

难以启齿的温柔 2024-07-19 08:54:27

我想这取决于你对它们做了什么......如果你要将它们的指标数据解析到数据库中,那么在每台机器上安装解析实用程序来解析并加载到你的中央数据库中会更快同一时间。

即使您所做的只是压缩并复制到中央位置,也可以在 .cmd 文件中设置这些命令并安排它在每台服务器上自动运行。 然后,您将在所有这些服务器之间分配工作,而不是强迫您的一个本地系统完成所有工作。 :-)

I guess it depends what you do with them ... if you are going to parse them for metrics data into a database, it would be faster to have that parsing utility installed on each of those machines to parse and load into your central database at the same time.

Even if all you are doing is compressing and copying to a central location, set up those commands in a .cmd file and schedule it to run on each of the servers automatically. Then you will have distributed the work amongst all those servers, rather than forcing your one local system to do all the work. :-)

我的黑色迷你裙 2024-07-19 08:54:27

我想到的第一个改进是不传送整个日志文件,而仅传送上次传送后的记录。 当然,这是假设文件随着时间的推移而积累,并且每次都不是全新的。

您可以通过多种方式实现这一点:如果文件具有您可以信赖的日期/时间戳,则通过过滤器运行它们,从考虑中删除较旧的记录并转储其余记录就足够了。 如果没有这样的鉴别器可用,我将跟踪发送的最后一个字节/行并在发货之前前进到该位置。

无论哪种方式,目标都是只发布新内容。 在我们自己的系统中,日志是通过一项服务传送的,该服务会在写入日志时复制日志。 这需要一个小型服务来处理要写入的日志文件,但可以减少捕获日志的延迟并极大地减少带宽使用。

The first improvement that comes to mind is to not ship entire log files, but only the records from after the last shipment. This of course is assuming that the files are being accumulated over time and are not entirely new each time.

You could implement this in various ways: if the files have date/time stamps you can rely on, running them through a filter that removes the older records from consideration and dumps the remainder would be sufficient. If there is no such discriminator available, I would keep track of the last byte/line sent and advance to that location prior to shipping.

Either way, the goal is to only ship new content. In our own system logs are shipped via a service that replicates the logs as they are written. That required a small service that handled the log files to be written, but reduced latency in capturing logs and cut bandwidth use immensely.

每个服务器可能应该:

  • 管理自己的日志文件(在上传之前启动新日志,并在上传后删除已发送的日志)
  • 为文件命名(或预先添加元数据),以便服务器知道哪个客户端发送了它们以及它们覆盖的时间段
  • 在发送之前压缩日志文件(压缩 + FTP + 解压缩通常比单独的 FTP 更快)
  • 将日志文件推送到中央位置(FTP 比 SMB 更快,Windows FTP 命令可以使用“-s:scriptfile”自动执行)
  • 当无法推送其日志时通知您出于任何原因,
  • 按交错时间表执行上述所有操作(以避免中央服务器超载)
    • 是否可以使用服务器的最后一个 IP 八位字节乘以一个常量来从午夜开始偏移(以分钟为单位)?

中央服务器可能应该:

  • 接受发送的日志文件并将其排队等待处理
  • 妥善处理接收同一日志文件两次(是否应该忽略或重新处理?)
  • 根据需要解压缩并处理日志文件
  • 根据保留策略删除/归档已处理的日志文件
  • 当服务器最近没有推送日志时通知您

Each server should probably:

  • manage its own log files (start new logs before uploading and delete sent logs after uploading)
  • name the files (or prepend metadata) so the server knows which client sent them and what period they cover
  • compress log files before shipping (compress + FTP + uncompress is often faster than FTP alone)
  • push log files to a central location (FTP is faster than SMB, the windows FTP command can be automated with "-s:scriptfile")
  • notify you when it cannot push its log for any reason
  • do all the above on a staggered schedule (to avoid overloading the central server)
    • Perhaps use the server's last IP octet multiplied by a constant to offset in minutes from midnight?

The central server should probably:

  • accept log files sent and queue them for processing
  • gracefully handle receiving the same log file twice (should it ignore or reprocess?)
  • uncompress and process the log files as necessary
  • delete/archive processed log files according to your retention policy
  • notify you when a server has not pushed its logs lately
温柔嚣张 2024-07-19 08:54:27

我们这里有一个较小规模的类似产品。 我们的解决方案是让生成日志文件的机器每天以随机交错的模式将它们推送到 NAT。 这解决了更多基于拉取的方法的许多问题,包括导致服务器忙碌数天的读写时间集中。

We have a similar product on a smaller scale here. Our solution is to have the machines generating the log files push them to a NAT on a daily basis in a randomly staggered pattern. This solved a lot of the problems of a more pull-based method, including bunched-up read-write times that kept a server busy for days.

国际总奸 2024-07-19 08:54:27

听起来存储服务器的带宽不会饱和,因此您可以并行地从不同位置的多个客户端提取数据。 主要问题是,减慢整个过程的瓶颈是什么?

It doesn't sound like the storage servers bandwidth would be saturated, so you could pull from several clients at different locations in parallel. The main question is, what is the bottleneck that slows the whole process down?

只是一片海 2024-07-19 08:54:27

我会执行以下操作:
编写一个程序在每个服务器上运行,它将执行以下操作:
监控服务器上的日志
按特定定义的时间表压缩它们
将信息传递到分析服务器。

编写位于核心服务器上的另一个程序,该程序执行以下操作:
当网络/CPU 不太忙时提取压缩文件。
(这可以是多线程的。)
这使用从终端计算机传递给它的信息来确定接下来要获取哪个日志。
解压缩并连续上传到您的数据库。

这应该为您提供一个解决方案,该解决方案可以提供最新信息,并且停机时间最短。
缺点是网络/计算机的使用相对一致,但这通常是一件好事。

它还可以轻松管理系统,检测任何需要解决的问题。

I would do the following:
Write a program to run on each server, which will do the following:
Monitor the logs on the server
Compress them at a particular defined schedule
Pass information to the analysis server.

Write another program which sits on the core srver which does the following:
Pulls compressed files when the network/cpu is not too busy.
(This can be multi-threaded.)
This uses the information passed to it from the end computers to determine which log to get next.
Uncompress and upload to your database continuously.

This should give you a solution which provides up to date information, with a minimum of downtime.
The downside will be relatively consistent network/computer use, but tbh that is often a good thing.

It will also allow easy management of the system, to detect any problems or issues which need resolving.

往事随风而去 2024-07-19 08:54:27

NetBIOS 副本不如 FTP 等快。 问题是您不希望每台服务器上都有 FTP 服务器。 如果您无法在每台服务器上本地处理日志文件,另一种解决方案是让所有服务器通过 FTP 将日志文件上传到一个中央位置,您可以从中进行处理。 例如:

设置 FTP 服务器作为中央收集点。 在每台服务器上安排任务以压缩日志文件并将存档通过 FTP 传输到中央 FTP 服务器。 您可以编写一个程序,使用 schtasks.exe 等工具远程自动安排任务:

KB 814596:如何在 Windows Server 2003 中使用 schtasks.exe 计划任务

您可能希望错开上传回 FTP 服务器的时间。

NetBIOS copies are not as fast as, say, FTP. The problem is that you don't want an FTP server on each server. If you can't process the log files locally on each server, another solution is to have all the server upload the log files via FTP to a central location, which you can process from. For instance:

Set up an FTP server as a central collection point. Schedule tasks on each server to zip up the log files and FTP the archives to your central FTP server. You can write a program which automates the scheduling of the tasks remotely using a tool like schtasks.exe:

KB 814596: How to use schtasks.exe to Schedule Tasks in Windows Server 2003

You'll likely want to stagger the uploads back to the FTP server.

~没有更多了~
我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站,即表示您同意使用 Cookies 和您的相关数据。
原文