当前位置：文江博客话题详情

如何在数据框架中删除表情符号和其他语言

发布于 2025-02-09 03:12:57 字数 120 浏览 4 评论 0 原文

我得到了一个数据框，如下所示：

标题内容

补水法

I got a dataframe as follows:

Title	Content
補水法????	Skin Care
???? 現貨 ????	รีบจัดด่วน‼️ ราคาเฉพาะรอบนี???? Test

I tried to use the regex:

df1['Post Title'] = df1['Post Title'].str.replace(r'[^\x00-\x7F]+', '', regex=True)
df1['Post Detail'] = df1['Post Detail'].str.replace(r'[^\x00-\x7F]+', '', regex=True)

It successfully removes the emojis. However, it only saves for the English but not the Chinese.

I would like to remove all the emojis and languages neither Chinese nor English.

Expected result:

Title	Content
補水法	Skin Care
現貨	Test

分享到QQ

分享到微博

如果你对这篇内容有疑问，欢迎到本站社区发帖提问参与讨论，获取更多帮助，或者扫码二维码加入 Web 技术交流群。

发布评论

需要登录才能够评论，你可以免费注册一个本站的账号。

萌能量女王 2025-02-16 03:12:58

这可以使用Python的Demoji库来完成。要使用PIP安装，

 pip install demoji

而不是

 import demoji
 df = df.applymap(lambda x: demoji.replace(x,''))

This can be done using the demoji library in python. For using do a pip install

 pip install demoji

Than

 import demoji
 df = df.applymap(lambda x: demoji.replace(x,''))

回复收藏 0 原文

江心雾 2025-02-16 03:12:58

我认为，该解决方案包含两个部分：

删除表情符号。表情符号可以通过Unicodes去除。详细的表情符号Unicode列表可以从。建议利用extessexages获取 emoji 之类的列表。例如，使用 emoji ，我们可以编写以下代码。 demoji 和 CleanText 之类的工具以类似的方式实现了他们的方法。

>>>

... from emoji import UNICODE_EMOJI

... df = pd.DataFrame({"Title": ["補水法

In my opinion, the solution contains two parts:

Remove emojis. Emojis can be removed by unicodes. The detailed emoji unicode list can be downloaded from http://www.unicode.org/emoji/charts/full-emoji-list.html. It is recommended to utilizing exsiting packages to obtain the list like emoji. For example, with emoji, we can write following code. Tools like demoji and cleantext implement their methods in a similar way.

>>>
... from emoji import UNICODE_EMOJI
... df = pd.DataFrame({"Title": ["補水法????", "???? 現貨 ????"], "Content": ["Skin Care", "รีบจัดด่วน‼️ ราคาเฉพาะรอบนี???? Test"]})
... def remove_emoji(text):
...     emojis = UNICODE_EMOJI["en"]
...     if isinstance(emojis, str):
...         emojis = [emojis]
...     result = text
...     for x in emojis:
...         result = result.replace(x, "")
...     return result
... df = df.applymap(lambda x: remove_emoji(x))
... print(df)
  Title                         Content
0   補水法                       Skin Care
1   現貨   รีบจัดด่วน ราคาเฉพาะรอบนี Test

Remove Non-English or Non-Chinese Characters. One possible way to do it is utilizing some language detection tools like langid, langdetect and so on. For example, with following code, the content with Thai language will be removed.

>>> 
... import langid
... df = pd.DataFrame({"Title": ["補水法????", "???? 現貨 ????"], "Content": ["Skin Care", "รีบจัดด่วน‼️ ราคาเฉพาะรอบนี???? Test"]})
... df = df.applymap(lambda x: '' if ((langid.classify(x)[0] != 'zh') and (langid.classify(x)[0] != 'en')) else x)

>>> print(df)

    Title    Content
0    補水法????  Skin Care
1  ???? 現貨 ????

If you want to acheive a more fine-grained result, say, you desire the result as below. I think you also need to tokenize the content. For Chinese tokenization, you can refer to https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/tokenizers/tokenizer_zh.py, which also includes unicode details of Hanzi.

  Title       Content
0   補水法    Skin Care
1   現貨      Test

回复收藏 0 原文

漫雪独思 2025-02-16 03:12:58

您可以提取所有英语和中文单词，并加入一个空间：

import pandas as pd

df = pd.DataFrame({'Title':['補水法

You can extract all English and Chinese words and join with a space:

import pandas as pd
df = pd.DataFrame({'Title':['補水法????', '???? 現貨 ????', 'Text, text'], 'Content':['Skin Care', 'รีบจัดด่วน‼️ ราคาเฉพาะรอบนี???? Test', 'More text!!!']})

pPunct = r'!-/\:-@\[-`\{-~\u00A1-\u00A9\u00AB\u00AC\u00AE-\u00B1\u00B4\u00B6-\u00B8\u00BB\u00BF\u00D7\u00F7\u02C2-\u02C5\u02D2-\u02DF\u02E5-\u02EB\u02ED\u02EF-\u02FF\u0375\u037E\u0384\u0385\u0387\u03F6\u0482\u055A-\u055F\u0589\u058A\u058D-\u058F\u05BE\u05C0\u05C3\u05C6\u05F3\u05F4\u0606-\u060F\u061B\u061D-\u061F\u066A-\u066D\u06D4\u06DE\u06E9\u06FD\u06FE\u0700-\u070D\u07F6-\u07F9\u07FE\u07FF\u0830-\u083E\u085E\u0888\u0964\u0965\u0970\u09F2\u09F3\u09FA\u09FB\u09FD\u0A76\u0AF0\u0AF1\u0B70\u0BF3-\u0BFA\u0C77\u0C7F\u0C84\u0D4F\u0D79\u0DF4\u0E3F\u0E4F\u0E5A\u0E5B\u0F01-\u0F17\u0F1A-\u0F1F\u0F34\u0F36\u0F38\u0F3A-\u0F3D\u0F85\u0FBE-\u0FC5\u0FC7-\u0FCC\u0FCE-\u0FDA\u104A-\u104F\u109E\u109F\u10FB\u1360-\u1368\u1390-\u1399\u1400\u166D\u166E\u169B\u169C\u16EB-\u16ED\u1735\u1736\u17D4-\u17D6\u17D8-\u17DB\u1800-\u180A\u1940\u1944\u1945\u19DE-\u19FF\u1A1E\u1A1F\u1AA0-\u1AA6\u1AA8-\u1AAD\u1B5A-\u1B6A\u1B74-\u1B7E\u1BFC-\u1BFF\u1C3B-\u1C3F\u1C7E\u1C7F\u1CC0-\u1CC7\u1CD3\u1FBD\u1FBF-\u1FC1\u1FCD-\u1FCF\u1FDD-\u1FDF\u1FED-\u1FEF\u1FFD\u1FFE\u2010-\u2027\u2030-\u205E\u207A-\u207E\u208A-\u208E\u20A0-\u20C0\u2100\u2101\u2103-\u2106\u2108\u2109\u2114\u2116-\u2118\u211E-\u2123\u2125\u2127\u2129\u212E\u213A\u213B\u2140-\u2144\u214A-\u214D\u214F\u218A\u218B\u2190-\u2426\u2440-\u244A\u249C-\u24E9\u2500-\u2775\u2794-\u2B73\u2B76-\u2B95\u2B97-\u2BFF\u2CE5-\u2CEA\u2CF9-\u2CFC\u2CFE\u2CFF\u2D70\u2E00-\u2E2E\u2E30-\u2E5D\u2E80-\u2E99\u2E9B-\u2EF3\u2F00-\u2FD5\u2FF0-\u2FFB\u3001-\u3004\u3008-\u3020\u3030\u3036\u3037\u303D-\u303F\u309B\u309C\u30A0\u30FB\u3190\u3191\u3196-\u319F\u31C0-\u31E3\u3200-\u321E\u322A-\u3247\u3250\u3260-\u327F\u328A-\u32B0\u32C0-\u33FF\u4DC0-\u4DFF\uA490-\uA4C6\uA4FE\uA4FF\uA60D-\uA60F\uA673\uA67E\uA6F2-\uA6F7\uA700-\uA716\uA720\uA721\uA789\uA78A\uA828-\uA82B\uA836-\uA839\uA874-\uA877\uA8CE\uA8CF\uA8F8-\uA8FA\uA8FC\uA92E\uA92F\uA95F\uA9C1-\uA9CD\uA9DE\uA9DF\uAA5C-\uAA5F\uAA77-\uAA79\uAADE\uAADF\uAAF0\uAAF1\uAB5B\uAB6A\uAB6B\uABEB\uFB29\uFBB2-\uFBC2\uFD3E-\uFD4F\uFDCF\uFDFC-\uFDFF\uFE10-\uFE19\uFE30-\uFE52\uFE54-\uFE66\uFE68-\uFE6B\uFF01-\uFF0F\uFF1A-\uFF20\uFF3B-\uFF40\uFF5B-\uFF65\uFFE0-\uFFE6\uFFE8-\uFFEE\uFFFC\uFFFD\U00010100-\U00010102\U00010137-\U0001013F\U00010179-\U00010189\U0001018C-\U0001018E\U00010190-\U0001019C\U000101A0\U000101D0-\U000101FC\U0001039F\U000103D0\U0001056F\U00010857\U00010877\U00010878\U0001091F\U0001093F\U00010A50-\U00010A58\U00010A7F\U00010AC8\U00010AF0-\U00010AF6\U00010B39-\U00010B3F\U00010B99-\U00010B9C\U00010EAD\U00010F55-\U00010F59\U00010F86-\U00010F89\U00011047-\U0001104D\U000110BB\U000110BC\U000110BE-\U000110C1\U00011140-\U00011143\U00011174\U00011175\U000111C5-\U000111C8\U000111CD\U000111DB\U000111DD-\U000111DF\U00011238-\U0001123D\U000112A9\U0001144B-\U0001144F\U0001145A\U0001145B\U0001145D\U000114C6\U000115C1-\U000115D7\U00011641-\U00011643\U00011660-\U0001166C\U000116B9\U0001173C-\U0001173F\U0001183B\U00011944-\U00011946\U000119E2\U00011A3F-\U00011A46\U00011A9A-\U00011A9C\U00011A9E-\U00011AA2\U00011C41-\U00011C45\U00011C70\U00011C71\U00011EF7\U00011EF8\U00011FD5-\U00011FF1\U00011FFF\U00012470-\U00012474\U00012FF1\U00012FF2\U00016A6E\U00016A6F\U00016AF5\U00016B37-\U00016B3F\U00016B44\U00016B45\U00016E97-\U00016E9A\U00016FE2\U0001BC9C\U0001BC9F\U0001CF50-\U0001CFC3\U0001D000-\U0001D0F5\U0001D100-\U0001D126\U0001D129-\U0001D164\U0001D16A-\U0001D16C\U0001D183\U0001D184\U0001D18C-\U0001D1A9\U0001D1AE-\U0001D1EA\U0001D200-\U0001D241\U0001D245\U0001D300-\U0001D356\U0001D6C1\U0001D6DB\U0001D6FB\U0001D715\U0001D735\U0001D74F\U0001D76F\U0001D789\U0001D7A9\U0001D7C3\U0001D800-\U0001D9FF\U0001DA37-\U0001DA3A\U0001DA6D-\U0001DA74\U0001DA76-\U0001DA83\U0001DA85-\U0001DA8B\U0001E14F\U0001E2FF\U0001E95E\U0001E95F\U0001ECAC\U0001ECB0\U0001ED2E\U0001EEF0\U0001EEF1\U0001F000-\U0001F02B\U0001F030-\U0001F093\U0001F0A0-\U0001F0AE\U0001F0B1-\U0001F0BF\U0001F0C1-\U0001F0CF\U0001F0D1-\U0001F0F5\U0001F10D-\U0001F1AD\U0001F1E6-\U0001F202\U0001F210-\U0001F23B\U0001F240-\U0001F248\U0001F250\U0001F251\U0001F260-\U0001F265\U0001F300-\U0001F6D7\U0001F6DD-\U0001F6EC\U0001F6F0-\U0001F6FC\U0001F700-\U0001F773\U0001F780-\U0001F7D8\U0001F7E0-\U0001F7EB\U0001F7F0\U0001F800-\U0001F80B\U0001F810-\U0001F847\U0001F850-\U0001F859\U0001F860-\U0001F887\U0001F890-\U0001F8AD\U0001F8B0\U0001F8B1\U0001F900-\U0001FA53\U0001FA60-\U0001FA6D\U0001FA70-\U0001FA74\U0001FA78-\U0001FA7C\U0001FA80-\U0001FA86\U0001FA90-\U0001FAAC\U0001FAB0-\U0001FABA\U0001FAC0-\U0001FAC5\U0001FAD0-\U0001FAD9\U0001FAE0-\U0001FAE7\U0001FAF0-\U0001FAF6\U0001FB00-\U0001FB92\U0001FB94-\U0001FBCA'
pLatin = r'A-Za-z\u00AA\u00BA\u00C0-\u00D6\u00D8-\u00F6\u00F8-\u02B8\u02E0-\u02E4\u1D00-\u1D25\u1D2C-\u1D5C\u1D62-\u1D65\u1D6B-\u1D77\u1D79-\u1DBE\u1E00-\u1EFF\u2071\u207F\u2090-\u209C\u212A\u212B\u2132\u214E\u2160-\u2188\u2C60-\u2C7F\uA722-\uA787\uA78B-\uA7CA\uA7D0\uA7D1\uA7D3\uA7D5-\uA7D9\uA7F2-\uA7FF\uAB30-\uAB5A\uAB5C-\uAB64\uAB66-\uAB69\uFB00-\uFB06\uFF21-\uFF3A\uFF41-\uFF5A\U00010780-\U00010785\U00010787-\U000107B0\U000107B2-\U000107BA\U0001DF00-\U0001DF1E'
pHan = r'\u2E80-\u2E99\u2E9B-\u2EF3\u2F00-\u2FD5\u3005\u3007\u3021-\u3029\u3038-\u303B\u3400-\u4DBF\u4E00-\u9FFF\uF900-\uFA6D\uFA70-\uFAD9\U00016FE2\U00016FE3\U00016FF0\U00016FF1\U00020000-\U0002A6DF\U0002A700-\U0002B738\U0002B740-\U0002B81D\U0002B820-\U0002CEA1\U0002CEB0-\U0002EBE0\U0002F800-\U0002FA1D\U00030000-\U0003134A'
pEmojiEx = r'0-9\u00A9\u00AE\u203C\u2049\u2122\u2139\u2194-\u2199\u21A9\u21AA\u231A\u231B\u2328\u23CF\u23E9-\u23F3\u23F8-\u23FA\u24C2\u25AA\u25AB\u25B6\u25C0\u25FB-\u25FE\u2600-\u2604\u260E\u2611\u2614\u2615\u2618\u261D\u2620\u2622\u2623\u2626\u262A\u262E\u262F\u2638-\u263A\u2640\u2642\u2648-\u2653\u265F\u2660\u2663\u2665\u2666\u2668\u267B\u267E\u267F\u2692-\u2697\u2699\u269B\u269C\u26A0\u26A1\u26A7\u26AA\u26AB\u26B0\u26B1\u26BD\u26BE\u26C4\u26C5\u26C8\u26CE\u26CF\u26D1\u26D3\u26D4\u26E9\u26EA\u26F0-\u26F5\u26F7-\u26FA\u26FD\u2702\u2705\u2708-\u270D\u270F\u2712\u2714\u2716\u271D\u2721\u2728\u2733\u2734\u2744\u2747\u274C\u274E\u2753-\u2755\u2757\u2763\u2764\u2795-\u2797\u27A1\u27B0\u27BF\u2934\u2935\u2B05-\u2B07\u2B1B\u2B1C\u2B50\u2B55\u3030\u303D\u3297\u3299\U0001F004\U0001F0CF\U0001F170\U0001F171\U0001F17E\U0001F17F\U0001F18E\U0001F191-\U0001F19A\U0001F1E6-\U0001F1FF\U0001F201\U0001F202\U0001F21A\U0001F22F\U0001F232-\U0001F23A\U0001F250\U0001F251\U0001F300-\U0001F321\U0001F324-\U0001F393\U0001F396\U0001F397\U0001F399-\U0001F39B\U0001F39E-\U0001F3F0\U0001F3F3-\U0001F3F5\U0001F3F7-\U0001F4FD\U0001F4FF-\U0001F53D\U0001F549-\U0001F54E\U0001F550-\U0001F567\U0001F56F\U0001F570\U0001F573-\U0001F57A\U0001F587\U0001F58A-\U0001F58D\U0001F590\U0001F595\U0001F596\U0001F5A4\U0001F5A5\U0001F5A8\U0001F5B1\U0001F5B2\U0001F5BC\U0001F5C2-\U0001F5C4\U0001F5D1-\U0001F5D3\U0001F5DC-\U0001F5DE\U0001F5E1\U0001F5E3\U0001F5E8\U0001F5EF\U0001F5F3\U0001F5FA-\U0001F64F\U0001F680-\U0001F6C5\U0001F6CB-\U0001F6D2\U0001F6D5-\U0001F6D7\U0001F6DD-\U0001F6E5\U0001F6E9\U0001F6EB\U0001F6EC\U0001F6F0\U0001F6F3-\U0001F6FC\U0001F7E0-\U0001F7EB\U0001F7F0\U0001F90C-\U0001F93A\U0001F93C-\U0001F945\U0001F947-\U0001F9FF\U0001FA70-\U0001FA74\U0001FA78-\U0001FA7C\U0001FA80-\U0001FA86\U0001FA90-\U0001FAAC\U0001FAB0-\U0001FABA\U0001FAC0-\U0001FAC5\U0001FAD0-\U0001FAD9\U0001FAE0-\U0001FAE7\U0001FAF0-\U0001FAF6'
pat = fr'[{pHan}{pLatin}{pPunct}]+'

df = df.replace(fr'[{pEmojiEx}]+', '', regex=True)
df['Title'] = df['Title'].str.findall(pat).str.join(" ")
df['Content'] = df['Content'].str.findall(pat).str.join(" ")
print(df.to_string())

Output:

>>> df
        Title       Content
0         補水法     Skin Care
1          現貨          Test
2  Text, text  More text!!!

What it does is:

df.replace(fr'[{pEmojiEx}]+', '', regex=True) removes all chars that are marked with Emoji Unicode property (these include digits, but I removed * and #)
.str.findall(fr'[{pHan}{pLatin}{pPunct}]+').str.join(" ") extracts chunks of one or more Latin, Han or any punctuation chars and joins them with a space.