当前位置： 文江博客话题详情

在 .NET 中以编程方式解析日志文件

发布于 2024-09-09 07:56:26 字数 4672 浏览 6 评论 0原文

我们有大量（读取：50,000）相对较小（读取低于 500K，通常低于 50K）的日志文件，这些日志文件是使用 log4net 从我们的客户端应用程序创建的。典型的日志如下所示：

Start Painless log
Framework:8.1.7.0
Application:8.1.7.0
2010-05-05 19:26:07,678 [Login ] INFO  Application.App.OnShowLoginMessage(194) - Validating Credentials...
2010-05-05 19:26:08,686 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Checking for Application Updates...
2010-05-05 19:26:08,830 [1     ] INFO  Framework.Globals.InstanceStartup(132) - Application Startup
2010-05-05 19:26:09,293 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Purchase History Data>:True
2010-05-05 19:26:09,293 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Shopping Assistant>:True
2010-05-05 19:26:09,294 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Shopping List>:True
2010-05-05 19:26:09,294 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Teeth>:True
2010-05-05 19:26:09,294 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Scanner>:True
2010-05-05 19:26:09,294 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Value Comparison>:True
2010-05-05 19:26:09,294 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Lotus Notes CRM>:True
2010-05-05 19:26:09,295 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Salesforce.com>:False
2010-05-05 19:26:09,295 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Lotus Notes Mail>:True
2010-05-05 19:26:09,295 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Sales Leads>:True
2010-05-05 19:26:09,295 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Configurator>:True
2010-05-05 19:26:09,297 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Validating Database...
2010-05-05 19:26:10,342 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Validating Database...
2010-05-05 19:26:10,489 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Global Handlers...
2010-05-05 19:26:10,495 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Starting Main Window...
2010-05-05 19:26:10,496 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Initializing Components...
2010-05-05 19:26:11,145 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Restoring Location...
2010-05-05 19:26:11,164 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Plug Ins...
2010-05-05 19:26:11,169 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Panels...Order Manager
2010-05-05 19:26:11,181 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Orders...
2010-05-05 19:26:11,274 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Done Loading 1 Order
2010-05-05 19:26:11,314 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Panels...All Products
2010-05-05 19:26:11,471 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Purchase History Data
2010-05-05 19:26:11,545 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Shopping List
2010-05-05 19:26:11,597 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Teeth
2010-05-05 19:26:11,768 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Scanner
2010-05-05 19:26:11,810 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Value Comparison
2010-05-05 19:26:11,840 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Sales Leads
2010-05-05 19:26:11,922 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...(Done!)
2010-05-05 19:26:11,923 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Adding Handlers...
2010-05-05 19:26:11,925 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Main Window Handlers...
2010-05-05 19:26:11,932 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Main Menu...
2010-05-05 19:26:11,949 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Initialization Complete.
2010-05-05 19:26:13,662 [1     ] INFO  Framework.ProductSearch.Search(342) - User entered term: <>

我希望能够解析这些日志服务器端（无论是在上传时还是在夜间）以提取异常（始终以 ERROR 或 FATAL 记录）或其他特定日志消息，例如：

2010-05-05 20:05:24,951 [1     ] INFO  Framework.ProductSearch.Search(342) - User entered term: <kavo>

获取“kavo”一词，以便我们可以找出人们真正搜索的内容。

我尝试过手动解析文本（使用 String.Split() 和类似的方法），但感觉就像是在重新发明轮子。

有没有一个不错的库来进行这种日志提取？

原文

We have a large number (read: 50,000) of relatively small (read under 500K, typically under 50K) log files created using log4net from our client application. A typical log looks like:

Start Painless log
Framework:8.1.7.0
Application:8.1.7.0
2010-05-05 19:26:07,678 [Login ] INFO  Application.App.OnShowLoginMessage(194) - Validating Credentials...
2010-05-05 19:26:08,686 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Checking for Application Updates...
2010-05-05 19:26:08,830 [1     ] INFO  Framework.Globals.InstanceStartup(132) - Application Startup
2010-05-05 19:26:09,293 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Purchase History Data>:True
2010-05-05 19:26:09,293 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Shopping Assistant>:True
2010-05-05 19:26:09,294 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Shopping List>:True
2010-05-05 19:26:09,294 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Teeth>:True
2010-05-05 19:26:09,294 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Scanner>:True
2010-05-05 19:26:09,294 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Value Comparison>:True
2010-05-05 19:26:09,294 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Lotus Notes CRM>:True
2010-05-05 19:26:09,295 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Salesforce.com>:False
2010-05-05 19:26:09,295 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Lotus Notes Mail>:True
2010-05-05 19:26:09,295 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Sales Leads>:True
2010-05-05 19:26:09,295 [1     ] INFO  Framework.PluginManager.LogPluginState(150) - Plugin <Configurator>:True
2010-05-05 19:26:09,297 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Validating Database...
2010-05-05 19:26:10,342 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Validating Database...
2010-05-05 19:26:10,489 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Global Handlers...
2010-05-05 19:26:10,495 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Starting Main Window...
2010-05-05 19:26:10,496 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Initializing Components...
2010-05-05 19:26:11,145 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Restoring Location...
2010-05-05 19:26:11,164 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Plug Ins...
2010-05-05 19:26:11,169 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Panels...Order Manager
2010-05-05 19:26:11,181 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Orders...
2010-05-05 19:26:11,274 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Done Loading 1 Order
2010-05-05 19:26:11,314 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Panels...All Products
2010-05-05 19:26:11,471 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Purchase History Data
2010-05-05 19:26:11,545 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Shopping List
2010-05-05 19:26:11,597 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Teeth
2010-05-05 19:26:11,768 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Scanner
2010-05-05 19:26:11,810 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Value Comparison
2010-05-05 19:26:11,840 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...Sales Leads
2010-05-05 19:26:11,922 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Tabbed Areas...(Done!)
2010-05-05 19:26:11,923 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Adding Handlers...
2010-05-05 19:26:11,925 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Main Window Handlers...
2010-05-05 19:26:11,932 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Loading Main Menu...
2010-05-05 19:26:11,949 [1     ] INFO  Application.App.OnShowLoginMessage(194) - Initialization Complete.
2010-05-05 19:26:13,662 [1     ] INFO  Framework.ProductSearch.Search(342) - User entered term: <>

I'd like to be able to to parse these logs server side (either when they're uploaded or nightly) to extract either exceptions (which always log at ERROR or FATAL) or other specific log messages like:

2010-05-05 20:05:24,951 [1     ] INFO  Framework.ProductSearch.Search(342) - User entered term: <kavo>

to get the 'kavo' term so we can find out what people are really searching for.

I've tried parsing the text by hand (using String.Split() and similar methods) but it really feels like I'm reinventing the wheel.

Is there a nice library to do this kind of log extracting?

如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

扫码二维码加入Web技术交流群

发布评论

需要 登录 才能够评论， 你可以免费 注册 一个本站的账号。

评论（4）

徒留西风 2024-09-16 07:56:26

您可以使用微软的 日志​​解析器 (在此处下载）。该工具还公开了一个 COM 接口，可让您以编程方式访问结果。 这篇博文使用 2.1 版执行此操作的小（部分）说明。

因此，现在您可能不会觉得它是在重新发明轮子，而是感觉它被过度设计了。 ;)

You could use Microsoft's Log Parser (Download here). The tool also exposes a COM interface that will let you access the results programmatically. This blog post has a small (partial) instructions for doing that with version 2.1.

So now instead of feeling it is reinventing the wheel, you probably feel like it is over-engineered. ;)

回复 原文
后eg是否自 2024-09-16 07:56:26

我们最终只是简单地输出 XML 作为我们的日志格式，并且解析变得很简单。

We ended up simply outputting XML as our log format and it became trivial to parse.

回复 原文
鱼忆七猫命九 2024-09-16 07:56:26

这看起来像是 .NET 正则表达式 的典型案例，

这样您就可以可以搜索诸如“用户输入的术语：”之类的模式。之后您可以提取小组赛

This looks like a typical case of .NET regular expressions

This way you can search for patterns like "User entered term: ". You can extract the group match afterwards

回复 原文
淡看悲欢离合 2024-09-16 07:56:26

log4net 不能读取它自己的格式模式吗？一旦它读回格式，你不能只解析 %message% 部分吗？

一个更好的问题可能是为什么您不只是在进行日志调用的同时将该搜索项写入数据库，或者通过 log4net 拥有另一个日志文件，该文件仅记录 %message% 并且您写入该日志的所有内容都是搜索学期？我真的不认为你想解析日志，如果你可以避免它......至少继续前进。

can log4net not read it's own format patterns? Once it reads the format back in, can't you then parse just the %message% section?

A better question maybe why you don't just write that search term to a database at the same time you make the log call or have another log file through log4net that just logs the %message% and all you write to that log is the search term? I really don't think you want to be parsing logs if you can avoid it...at least going forward.

回复 原文
~没有更多了~

关于作者

禾厶谷欠

暂无简介

0 文章
0 评论
25 人气
发私信

相关话题

更多

热门标签

操作系统 程序设计 IT运维 Linux系统管理 JavaScript 服务器应用 solaris C/C++ PHP Shell BSD Vue.js aix Oracle Python HTML 系统管理 HTML5 CSS 前端
更多

推荐作者

娇女薄笑

文章 0 评论 0

biaggi

文章 0 评论 0

xiaolangfanhua

文章 0 评论 0

rivulet

文章 0 评论 0

我三岁

文章 0 评论 0

薆情海

文章 0 评论 0

更多

友情链接

文江博客
    我们使用 Cookies 和其他技术来定制您的体验包括您的登录状态等。通过阅读我们的 隐私政策 了解更多相关信息。 单击 接受 或继续使用网站，即表示您同意使用 Cookies 和您的相关数据。
    原文