在用 Python 编写之前检查 txt 文件中的匹配项

Question

我正在处理一个非常大的文本文件（500MB+），我的代码输出完美，但我得到了很多重复项。我想要做的是在写入文件之前检查输出文件以查看输出是否存在。我确信这只是 if 语句中的一行，但我不太了解 python，无法弄清楚语法。任何帮助将不胜感激。

这是代码：

authorList = ['Shakes.','Scott']

with open('/Users/Adam/Desktop/Poetrylist.txt','w') as output_file:
    with open('/Users/Adam/Desktop/2e.txt','r') as open_file:
            the_whole_file = open_file.read()
            for x in authorList:
                start_position = 0 
                while True:
                   start_position = the_whole_file.find('<A>'+x+'</A>', start_position)
                   if start_position < 0:
                       break
                   end_position = the_whole_file.find('</W>', start_position)
                   output_file.write(the_whole_file[start_position:end_position+4])
                   output_file.write("\n")    
                   start_position = end_position + 4

Answer 1

我建议你简单地记录下你已经看过的作者数据，只有在你以前没有看过的情况下才写下来。您可以使用dict来跟踪。

authorList = ['Shakes.','Scott']
already_seen = {} # dict to keep track of what has been seen

with open('/Users/Adam/Desktop/Poetrylist.txt','w') as output_file:
    with open('/Users/Adam/Desktop/2e.txt','r') as open_file:
            the_whole_file = open_file.read()
            for x in authorList:
                start_position = 0 
                while True:
                   start_position = the_whole_file.find('<A>'+x+'</A>', start_position)
                   if start_position < 0:
                       break
                   end_position = the_whole_file.find('</W>', start_position)
                   author_data = the_whole_file[start_position:end_position+4]
                   if author_data not in already_seen:
                       output_file.write(author_data + "\n")
                       already_seen[author_data] = True
                   start_position = end_position + 4

Answer 2

创建一个列表，其中包含要写入的每个字符串。如果您追加它，请首先检查您追加的项目是否已在列表中。

Answer 3

我的理解是，当您想要写入output_file时，您希望跳过open_file中包含作者姓名的行。如果这是您打算做的，那么就这样做。

authorList = ['Shakes.','Scott']

with open('/Users/Adam/Desktop/Poetrylist.txt','w') as output_file:
    with open('/Users/Adam/Desktop/2e.txt','r') as open_file:
         for line in open_file:
              skip = 0
              for author in authorList:
                   if author in line:
                       skip = 1
              if not skip:
                   output_file.write(line)

Answer 4

我认为您应该使用适当的工具来处理您的文件来处理文本：正则表达式。

import re

regx = re.compile('<A>(.+?)</A>.*?<W>.*?</W>')

with open('/Users/Desktop/2e.txt','rb')         as open_file,\
     open('/Users/Desktop/Poetrylist.txt','wb') as output_file:

    remain = ''
    seen = set()

    while True:
        chunk = open_file.read(65536) # 65536 == 16 x 16 x 16 x 16
        if not chunk:  break
        for mat in regx.finditer(remain + chunk):
            if mat.group(1) not in seen:
                output_file.write( mat.group() + '\n' )
                seen.add(mat.group(1))
        remain = chunk[mat.end(0)-len(remain):]