Python从文件中读取文本未找到撇号

Question

用于清理文本的函数

def clean_before_tok(text):
    text=text.replace("'"," ")
    exclude=[" le "," la "," l "," un "," une "," du "," de "," les "," des "," s "," d "]
    for e in exclude:
        text=text.replace(e," ")
    return text

我可以在宠物示例上测试它

test=clean_before_tok("dlkj dfg le se d'ac")
print(test)
>>> dlkj dfg se ac

但是当从文件中读取时

generated_text=open("text-like.txt", 'rb').read().decode(encoding='utf-8')

它没有找到替换撇号。是否存在编码缺陷？

Answer 1

为了检查文件的编码，您可以将其打印为字节

>>> with open("my-file.txt", "rb") as file:
...     b_file = file.read()
>>> print(b_file)

如果撇号显示为撇号，那就很奇怪了。通常，该问题可以通过文本中出现奇怪的 \xAB（AB 可以是任何大写或小写字母，它们代表非 ASCII 字节）来解释。