Force character vector encoding from “unknown” to “UTF-8” in R

by

The Encoding function returns unknown if a character string has a “native encoding” mark (CP-1250 in your case) or if it’s in ASCII.
To discriminate between these two cases, call:

library(stringi)
stri_enc_mark(poli.dt$word)

To check whether each string is a valid UTF-8 byte sequence, call:

all(stri_enc_isutf8(poli.dt$word))

If it’s not the case, your file is definitely not in UTF-8.

I suspect that you haven’t forced the UTF-8 mode in the data read function (try inspecting the contents of poli.dt$word to verify this statement). If my guess is true, try:

read.csv2(file("filename", encoding="UTF-8"))

or

poli.dt$word <- stri_encode(poli.dt$word, "", "UTF-8") # re-mark encodings

If data.table still complains about the “mixed” encodings, you may want to transliterate the non-ASCII characters, e.g.:

stri_trans_general("Zażółć gęślą jaźń", "Latin-ASCII")
## [1] "Zazolc gesla jazn"

More Related Contents:

  1. Read a UTF-8 text file with BOM
  2. Manipulating files with non-English names in R
  3. Naming columns in a data table in R
  4. How to use Greek symbols in ggplot2?
  5. data.table objects assigned with := from within function not printed
  6. What is the purpose of setting a key in data.table?
  7. How to efficiently calculate distance between pair of coordinates using data.table :=
  8. Sample random rows within each group in a data.table
  9. How to perform join over date ranges using data.table?
  10. Split text string in a data.table columns
  11. how to read data in utf-8 format in R?
  12. Update subset of data.table based on join
  13. Select values from different columns based on a variable containing column names [duplicate]
  14. Fetching UTF-8 text from MySQL in R returns “????”
  15. How to speed up subset by groups
  16. knitr gets tricked by data.table `:=` assignment
  17. Add multiple columns to R data.table in one function call?
  18. Extract a column from a data.table as a vector, by position
  19. data.table merge based on date ranges
  20. Reason behind speed of fread in data.table package in R
  21. data.table row-wise sum, mean, min, max like dplyr?
  22. := (pass by reference) operator in the data.table package modifies another data table object simultaneously
  23. Rowwise cumulative sum
  24. Convert a data frame to a data.table without copy
  25. Row sums over columns with a certain pattern in their name
  26. Data Table – Select Value of Column by Name From Another Column
  27. paste two data.table columns
  28. Removing multiple columns from R data.table with parameter for columns to remove
  29. Merging multiple data.tables
  30. Fill option for fread

Leave a Comment