Question

新的限制会产生新的形式。毫无例外，推特就属于这一种。因为文本必须合乎140个字符的限制，人们自然就开发出了新的语言简写形式，用更少的字符来说同样的事情。但到目前为止，我们忽略了这些多种多样的表情和缩写。让我们看看如果把这些也考虑进去的话，将会带来多少提升。对此，我们将会为TfidfVectorizer 提供自己定制的preprocessor() 。

首先，我们在一个字典中定义了一系列常用表情和它们的替代词语。尽管可以找到更多的替代词语，但我们只采用那些明显带有正面或负面情感的词语来帮助分类器：

emo_repl = { # 正面情感的表情 "<3": " good ", ":d": " good ", # 小写的:D ":dd": " good ", # 小写的 :DD "8)": " good ", ":-)": " good ", ":)": " good ", ";)": " good ", "(-:": " good ", "(:": " good ", # 负面情感的表情 ":/": " bad ", ":>": " sad ", ":')": " sad ", ":-(": " bad ", ":(": " bad ", ":S": " bad ", ":-S": " bad ", } # 确保:dd在:d之前被替代 emo_repl_order = [k for (k_len,k) in reversed(sorted([(len(k),k) for k in emo_repl.keys()]))]

然后，我们用正则表达式及其扩展（\b 标记出词语边界）来定义那些缩写形式：

re_repl = { r"\br\b": "are", r"\bu\b": "you", r"\bhaha\b": "ha", r"\bhahaha\b": "ha", r"\bdon't\b": "dnot", r"\bdoesn't\b": "does not", r"\bdidn't\b": "did not", r"\bhasn't\b": "has not", r"\bhaven't\b": "have not", r"\bhadn't\b": "had not", r"\bwon't\b": "will not", r"\bwouldn't\b": "would not", r"\bcan't\b": "can not", r"\bcannot\b": "can not", } def create_ngram_model(params=None): def preprocessor(tweet): global emoticons_replaced tweet = tweet.lower() #返回 tweet.lower() for k in emo_repl_order: tweet = tweet.replace(k, emo_repl[k]) for r, repl in re_repl.iteritems(): tweet = re.sub(r, repl, tweet) return tweet tfidf_ngrams = TfidfVectorizer(preprocessor=preprocessor, analyzer="word") # ...

当然，还有更多的缩写可以在这里使用。但就是使用这个有限的集合，我们已经在“有情感的推文 vs. 无半点情感的推文”的分类中得到了效果提升，达到了70.7%：

== Pos vs. neg == 0.804 0.022 0.886 0.011 == Pos/neg vs. irrelevant/neutral == 0.797 0.009 0.707 0.029 == Pos vs. rest == 0.884 0.005 0.527 0.025 == Neg vs. rest == 0.886 0.011 0.640 0.032