一些法语重音字符被编码为 utf-8 但仍然无法正确呈现

Question

您好：我正在导入一个包含大量法语口音字符的 Stata 文件。导入时，我将编码设置为 utf-8。但是，某些重音字符无法正确呈现。请参阅下面我的数据集中的行示例。 我该如何处理这个问题？

test<-tibble::tribble(
  ~municipality,
  "Sainte-Anne-de-Beaupré",
  "Sainte-Anne-de-Beaupré",
  "Sainte-Anne-de-Beaupré",
  "Beaupré",
  "Beaupré",
  "Beaupré",
  "Beaupré",
  "Beaupré",
  "Beaupré"
)
Encoding(test$municipality)
Encoding(test$municipality)<-'utf-8'
test$municipality

Answer 1

正如 Giacomo 提到的，这似乎是文件的一个示例，其中一部分（正如您还显示了正确编码的 é，其中 UTF-8 一样，将其读取为 Latin1 并再次编码为 UTF-8，这意味着您的编码是正确的 à 并且这样显示。

知道它是如何发生的意味着我们知道如何修复它！

© 本身也是 utf-8 字符， 重大失误三次错误的编码），因此我们模拟每个字符在错误编码后的变化，然后再次保存为 utf8

FixEncoding <- function() {
  # create the unicode ranges from https://www.i18nqa.com/debug/utf8-debug.html
  range <- c(sprintf("%x", seq(strtoi("0xa0"), strtoi("0xff"))))
  unicode <- vapply(range, FUN.VALUE = character(1), function(x) { parse(text = paste0("'\\u00", x, "'"))[[1]] })
  # add the ones that are missing (red ones in https://www.i18nqa.com/debug/utf8-debug.html)
  unicode <- c(c("\u0168", "\u0152", "\u017d", "\u0153", "\u017e", "\u0178", "\u2019", "\u20ac", "\u201a", "\u0192", "\u201e", "\u2026", "\u2020", "\u2021", "\u02c6", "\u2030", "\u0160", "\u2030"), unicode)
  once <- vapply(unicode, FUN.VALUE = character(1), function(x) { 
    Encoding(x) <- "Windows-1252"
    iconv(x, to = "UTF-8")
  })
  fix_once <- unicode
  names(fix_once) <- once
  twice <- vapply(once, FUN.VALUE = character(1), function(x) { 
    Encoding(x) <- "Windows-1252"
    iconv(x, to = "UTF-8")
  })
  fix_twice <- unicode
  names(fix_twice) <- twice
  triple <- vapply(twice, FUN.VALUE = character(1), function(x) { 
    Encoding(x) <- "Windows-1252"
    iconv(x, to = "UTF-8")
  })
  fix_triple <- unicode
  names(fix_triple) <- triple
  fixes <- c(fix_triple, fix_twice, fix_once)
  return(fixes)
}

fixes <- FixEncoding()

让我们在您的数据上运行它

v <- c("Sainte-Anne-de-Beaupré", "Beaupré")
v
# [1] "Sainte-Anne-de-Beaupré" "Beaupré"

stringi::stri_replace_all_fixed(v, names(fixes), fixes, vectorize_all = F)
# [1] "Sainte-Anne-de-Beaupré" "Beaupré"

另一个例子：

str <- "\u006C\u00E9\u006C\u00E8\u006C\u00F6\u006C\u00E3"
str
# [1] "lélèlölã"

# how to corrupt it once
Encoding(str) <- "Windows-1252"
str <- iconv(str, to = "UTF-8")
str
# [1] "lélèlölã"

# add once, twice and tripple wrong encoded string to messy
messy <- c("lélèlölã", "lélèlölã", "lélèlölã")

# All three strings above would be fixed
stri_replace_all_fixed(messy, names(fixes), fixes, vectorize_all = F)
# [1] "lélèlölã" "lélèlölã" "lélèlölã"