Unicodedecodeerror：＆＃x27; utf-8＆＃x27;编解码器可以在位置139390中解码字节0xe1：从Twitter API刮擦时无效的连续字节

Question

我有一个使用python软件包构建的网络拖纸器，我始终使用它来收集推文以进行研究。突然，它似乎不再起作用了。问题是不能再解码所有字符吗？

# open and create a file to append the data to
csvFile = open('tweets.csv', 'a')
csvWriter = csv.writer(csvFile)
    # use the csv file
    # loop through the tweets variable and add contents to the CSV file
for tweet in tweets:
    text = tweet.full_text.strip()
    #convert the text to ascii ignoring all unicode characters, eg. emojis
    text_ascii = text.encode('ascii','ignore').decode()
    #split the text on whitespace and newlines into a list of words
    text_list = text_ascii.split()
    #iterate over the words, removing @ mentions or URLs 
    text_list_filtered = [word for word in text_list if not (word.startswith('@') or word.startswith('http'))]
    #join the list back into a string
    text_filtered = ' '.join(text_list_filtered)
    #decoding html escaped characters
    text_filtered = html.unescape(text_filtered)
    #write text to the CSV file
    csvWriter.writerow([tweet.created_at, tweet.place, text_filtered])
    print(tweet.created_at, tweet.place, text_filtered)
csvFile.close() 

因此，当我尝试将其阅读为Pandas DataFrame时，我会收到此错误：

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xe1 in position 139390: invalid continuation byte

行是：

tweetsdf = pd.read_csv('tweets.csv')

我试图将以下代码更改为

text_ascii = text.encode('ascii','ignore').decode()

：

text_ascii = text.encode('utf-8','ignore').decode()

给我错误的 当我尝试从API收集推文时。我应该怎么办？

Answer 1

E1字节似乎是由以下事实引起的：对于某些推文，代码打印了发布推文的位置。从“ csvwriter.writerow（[[Tweet.created_at，Tweet.place，text_filtered text_filtered）删除'tweet.place'之后再次将CSV文件放入PANDAS DataFrame。