优化这个python日志解析代码
在我的笔记本电脑上，此代码对于 4.2 GB 输入文件的运行时间为 48 秒。输入文件以制表符分隔，每个值都用引号引起来。每个记录都以换行符结尾，例如 '"val1"\t"val2"\t"val3"\t..."valn"\n'
我尝试过使用 10 个线程的多处理： 1 个用于对行进行排队，8 个用于解析各个行并填充输出队列，还有一个用于将输出队列减少到如下所示的 defaultdict 中，但代码运行时间为 300 秒，比以下代码长 6 倍多：
from collections import defaultdict
def get_users(log):
users = defaultdict(int)
f = open(log)
# Read header line
h = f.readline().strip().replace('"', '').split('\t')
ix_profile = h.index('profile.type')
ix_user = h.index('profile.id')
# If either ix_* is the last field in h, it will include a newline.
# That's fine for now.
for (i, line) in enumerate(f):
if i % 1000000 == 0: print "Line %d" % i # progress notification
l = line.split('\t')
if l[ix_profile] != '"7"': # "7" indicates a bad value
# use list slicing to remove quotes
users[l[ix_user][1:-1]] += 1
f.close()
return users
I'我已经检查过我是通过从 for 循环中删除除 print 语句之外的所有内容，不受 I/O 限制。该代码在 9 秒内运行，我将考虑该代码运行速度的下限。
我有很多这样的 5 GB 文件需要处理，因此即使运行时的一个很小的改进（我知道，我可以删除打印！）也会有所帮助。我运行的机器有 4 个核心，所以我忍不住想知道是否有办法让多线程/多进程代码运行得比上面的代码更快。
更新：
我重写了多处理代码，如下：
from multiprocessing import Pool, cpu_count
from collections import defaultdict
def parse(line, ix_profile=10, ix_user=9):
"""ix_profile and ix_user predetermined; hard-coding for expedience."""
l = line.split('\t')
if l[ix_profile] != '"7"':
return l[ix_user][1:-1]
def get_users_mp():
f = open('20110201.txt')
h = f.readline() # remove header line
pool = Pool(processes=cpu_count())
result_iter = pool.imap_unordered(parse, f, 100)
users = defaultdict(int)
for r in result_iter:
if r is not None:
users[r] += 1
return users
它在 26 秒内运行，加速了 1.85 倍。还不错，但是有 4 个核心，没有我希望的那么好。
The runtime of this code on my laptop for a 4.2 GB input file is 48 seconds. The input file is tab-delimited, with each value appearing in quotes. Each record ends with a newline, e.g. '"val1"\t"val2"\t"val3"\t..."valn"\n'
I've tried using multiprocessing with 10 threads: One to queue up the lines, 8 to parse individual lines and fill an output queue, and one to reduce the output queue into the defaultdict shown below, but the code took 300 seconds to run, over 6 times longer than the following:
from collections import defaultdict
def get_users(log):
users = defaultdict(int)
f = open(log)
# Read header line
h = f.readline().strip().replace('"', '').split('\t')
ix_profile = h.index('profile.type')
ix_user = h.index('profile.id')
# If either ix_* is the last field in h, it will include a newline.
# That's fine for now.
for (i, line) in enumerate(f):
if i % 1000000 == 0: print "Line %d" % i # progress notification
l = line.split('\t')
if l[ix_profile] != '"7"': # "7" indicates a bad value
# use list slicing to remove quotes
users[l[ix_user][1:-1]] += 1
f.close()
return users
I've checked that I'm not I/O-bound by removing everything but the print statement from the for loop. That code ran in 9 seconds, which I'll consider a lower-bound for how fast this code can run.
I have a lot of these 5 GB files to process, so even a pretty small improvement in runtime (I know, I can remove the print!) will help. The machine I am running on has 4 cores, so I can't help but wonder if there's a way to get the multithread/multiprocess code to run faster than the code above.
UPDATE:
I rewrote the multiprocessing code as follows:
from multiprocessing import Pool, cpu_count
from collections import defaultdict
def parse(line, ix_profile=10, ix_user=9):
"""ix_profile and ix_user predetermined; hard-coding for expedience."""
l = line.split('\t')
if l[ix_profile] != '"7"':
return l[ix_user][1:-1]
def get_users_mp():
f = open('20110201.txt')
h = f.readline() # remove header line
pool = Pool(processes=cpu_count())
result_iter = pool.imap_unordered(parse, f, 100)
users = defaultdict(int)
for r in result_iter:
if r is not None:
users[r] += 1
return users
It runs in 26 seconds, a 1.85x speedup. Not bad, but with 4 cores, not as much as I had hoped for.
如果你对这篇内容有疑问，欢迎到本站社区发帖提问 参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。
绑定邮箱获取回复消息
由于您还没有绑定你的真实邮箱，如果其他用户或者作者回复了您的评论，将不能在第一时间通知您！
发布评论
评论（7）
使用正则表达式。
测试确定该过程中最昂贵的部分是对 str.split() 的调用。可能必须为每一行构造一个列表和一堆字符串对象，成本很高。
首先，您需要构造一个正则表达式来匹配该行。类似于：
如果您调用 expression.match(line).groups()，您将获得前两列提取为两个字符串对象，并且您可以直接对它们进行逻辑处理。
现在假设感兴趣的两列是前两列。如果没有，您只需调整正则表达式以匹配正确的列。您的代码检查标题以查看列所在的位置。您可以基于此生成正则表达式，但我猜这些列实际上总是位于同一位置。只需验证它们是否仍然存在并在行上使用正则表达式即可。
编辑
from collections import defaultdict
import re
此代码假设用户位于 profile 之前
此正则表达式将捕获单个列
此正则表达式将匹配该列，但不捕获结果。 （注意缺少括号）
为所有字段创建一个列表，只保留我们关心的字段
删除我们关心的字段后面的所有字段（它们需要时间来匹配，我们不关心它们）
实际上生成正则表达式。
取出两个值，并进行逻辑运算
Use regular expressions.
A test determines that the expensive part of the process is the call to str.split(). Probably having to construct a list and a bunch of string objects for every line is expensive.
Firstly, you need to construct a regular expression to match against the line. Something like:
If you call expression.match(line).groups(), you'll get the first two columns extracted as two string objects and you can do logic with those directly.
Now this assumes that the two columns of interest are the first two columns. If not you'll just have to tweak the regular expression to match the correct columns. Your code checks the header to see where the columns are located. You can generate the regular expression based on that, but I'm gonna guess that the columns are really always located at the same place. Just verify that they are still there and use a regular expression on the lines.
EDIT
from collections import defaultdict
import re
This code assumes that user is before profile
This regular expression will capture a single column
This regular expression will match the column, but not capture the results. (Note the lack of parenthesis)
Create a list for all the fields, and only the keep the ones we care about
Eliminate all the fields after the ones we care about (they take time to match, and we don't care about them)
Actually produce the regex.
Pull out the two values, and do the logic
如果您运行的是 unix 或 cygwin，下面的小脚本将为您生成用户 ID 的频率，其中 profile != 7。应该很快。
使用 awk 更新以计算用户 ID
If you're running unix or cygwin, the following little script would produce you the frequency of user id's where profile != 7. Should be quick.
Updated with awk to count the user ids
看到您的日志文件是制表符分隔的，您可以使用
csv模块 - 带有dialect='excel-tab'参数 - 以获得良好的性能和可读性提升。当然，如果您必须使用 Python 而不是更快的控制台命令的话。Seeing that your log file is tab-delimited, you can use the
csvmodule - with adialect='excel-tab'argument - for a nice performance and readability boost. That is, of course, if you have to use Python instead of the much faster console commands.如果使用正则表达式可以通过忽略不需要拆分的行尾部来加快速度，也许更直接的方法可能会有所帮助：
请对您的数据进行旋转。
If using regexes can speed it up so much by ignoring the tail of the line that doesn't need to be split, perhaps a more straightforward approach might help:
Please give that a whirl on your data.
我意识到我的想法与 Winston Ewert 几乎完全相同：构建一个正则表达式。
但我的正则表达式：
是针对 ix_profile
ix_profile的情况完成的。 ix_user以及 ix_profile > 的情况ix_user正则表达式仅捕获用户的列：配置文件的列与子模式匹配
'"(?!7")[^\t\r\n"]*"'如果此列中存在 "7"，则不匹配；因此，我们仅获取定义了唯一组的正确用户。
此外，我还测试了几种匹配和提取算法：
1）使用 re.finditer()
2) 使用 re.match() 以及正则表达式匹配 40 个字段
3) 使用 re.match() 且正则表达式仅匹配 max(ix_profile,ix_user) + 1 个字段
4) like 3 但有一个简单字典而不是 defaultdict 实例
为了测量时间，我的代码根据您提供的有关其内容的信息创建一个文件。
。
我在 4 个代码中测试了以下 4 个函数：
1
2
3
4
完整的代码 4，这似乎是最快的：
该代码 4 的一个结果是例如：
但结果或多或少是可变的。我得到：
。
我想知道您使用我的代码在您的真实文件上使用肯定比我更强大的计算机会获得什么样的结果。请给我消息。
。
。
编辑 1
我
已经
使用了一个简单的字典：
提高了一点执行时间，但它仍然高于我最后的代码 4
。
编辑2
最快：
结果
。
编辑 3
我测试了 get_users_short_CSV() 文件中包含 10000 行而不是仅 1000 行：
I realize that I had nearly exactly the same idea than Winston Ewert: building a regex.
But my regex:
is done to the cases in which
ix_profile < ix_useras well the cases in whichix_profile > ix_userthe regex captures only the user's column: the profile's column is matched with a sub-pattern
'"(?!7")[^\t\r\n"]*"'that doesn't match if "7" is present in this column; so we obtain only the correct user with the only group defined.
Additionally, I tested several algorithms of matching and extracting:
1) with re.finditer()
2) with re.match() and the regex matching 40 fields
3) withe re.match() and the regex matching only max(ix_profile,ix_user) + 1 fields
4) like 3 but with a simple dictionary instead of a defaultdict instance
To measure times, my code creates a file based on the information you gave concerning its content.
.
I tested the 4 following functions in 4 codes:
1
2
3
4
The full code 4, that seems to be the fastest:
One result of this code 4 is for exemple:
But results are more or less variable. I obtained:
.
I'd like to know what kind of result you would obtain with my code on your real file with a computer certainly more powerful than mine. Please, give me news.
.
.
EDIT 1
With
I've got
Using a simple dict:
improves a little the execution's time but it remains higher than my last code 4
.
EDIT 2
The fastest:
result
.
EDIT 3
I tested get_users_short_CSV() with 10000 lines in the file instead of only 1000:
也许你可以
代替
并删除最后字典上的引号。应该可以节省一些时间。
对于多线程方法：尝试每次从文件中读取几千行并将这几千行传递给线程进行处理。逐行进行似乎开销太大。
或者阅读本文中的解决方案他似乎正在做的事情与你想做的事情非常相似。
Maybe you can do
instead of
and remove the quotes on the dict at the end. Should save some time.
For the multi-threading approach: try reading a few thousand lines from the file each time and passing those thousand lines to a thread to process. Doing it line-by-line seems to be too much overhead.
Or read on the solution in this article as he seems to be doing something very similar to what you're trying to do.
这可能有点离题，但 Python 在处理多个线程时有一些极其奇怪的行为（当线程不受 IO 限制时尤其糟糕）。更具体地说，它有时比单线程时运行得慢得多。这是由于 Python 中使用全局解释器锁 (GIL) 的方式来确保在任何给定时间，Python 解释器中只能执行一个线程。
由于在任何给定时间只有一个线程可以实际使用解释器的限制，因此拥有多个内核这一事实对您没有帮助。事实上，由于试图获取 GIL 的两个线程之间的一些病态交互，实际上可能会让事情变得更糟。如果您想坚持使用 Python，您有以下两个选择之一：
print语句。如果您想了解有关 Python 的这个极其神秘的部分的更多信息，请查看此页面上与 GIL 相关的演讲：http ://www.dabeaz.com/talks.html。
This may be slightly besides the point, but Python has some extremely strange behavior when dealing with multiple threads (particularly bad when the threads aren't IO bound). More specifically, it sometimes runs much slower than when single-threaded. This is due to the way that the Global Interpreter Lock (GIL) in Python gets used to ensure that no more than one thread can execute in the Python interpreter at any given time.
Because of the constraint that only one thread can actually use the interpreter at any given time, the fact that you have multiple cores won't help you. In fact, it might actually make things much worse due to some pathological interactions between two threads trying to acquire the GIL. If you want to stick to Python you have one of two options:
printstatement.If you want more information on this wonderfully arcane bit of Python look up the talks related to the GIL on this page: http://www.dabeaz.com/talks.html.