R on Windows: character encoding hell

by

It is possible that your problem is solved by changing fileEncoding into encoding, these parameters work differently in the read function (see ?read).

oem.csv <- read.table("~/csv1.csv", sep=";", dec=",", quote="",encoding="cp866")

Just in case however, a more complete answer, as there may be some non-obvious obstacles. In short: It is possible to work with Cyrillic in R on Windows (in my case Win 7).

You may need to try a few possible encodings to get things to work. For text mining, an important aspect is to get the your input variables to match the data. There the function of Encoding() is very useful, see also iconv(). Thus it is possible to see your native parameter.

Encoding(variant <- "Минемум")

In my case the encoding is UTF-8, although this may depend on system settings. So, we can try the result with UTF-8 and UTF-8-BOM, and make a test file in notepad++ with a line of Latin and a line of Cyrillic.

UTF8_nobom_cyrillic.csv & UTF8_bom_cyrillic.csv

part2, part3, part4
Минемум конкыптам, тхэопхражтуз, ед про

This can be imported into R by 

raw_table1 <- read.csv("UTF8_nobom_cyrillic.csv", header = FALSE, sep = ",", quote = "\"", dec = ".", fill = TRUE, comment.char = "", encoding = "UTF-8")
raw_table2 <- read.csv("UTF8_bom_cyrillic.csv", header = FALSE, sep = ",", quote = "\"", dec = ".", fill = TRUE, comment.char = "", encoding = "UTF-8-BOM")

The results of these are for me for BOM regular Cyrillic in the view(raw_table1), and gibberish in console.

part2, part3, part4
ŠŠøŠ½ŠµŠ¼ŃŠ¼ ŠŗŠ¾Š½ŠŗŃ‹ŠæŃ‚Š°Š¼ тхѨŠ¾ŠæŃ…Ń€Š°Š¶Ń‚ŃŠ

More importantly however, the script does not give access to it.

> grep("Минемум", as.character(raw_table2[2,1]))
integer(0)

The results for No BOM UTF-8, are something like this for both view(raw_table1) and console.

part2, part3, part4
<U+041C><U+0438><U+043D><U+0435><U+043C><U+0443><U+043C> <U+043A><U+043E><U+043D><U+043A><U+044B><U+043F><U+0442><U+0430><U+043C> <U+0442><U+0445><U+044D><U+043E><U+043F><U+0445><U+0440><U+0430><U+0436><U+0442><U+0443><U+0437> <U+0435><U+0434> <U+043F><U+0440><U+043E>

However, importantly, a search for the word inside will yield the correct result.

> grep("Минемум", as.character(raw_table1[2,1]))
1

Thus, it is possible to work with non-standard characters in Windows, depending on your precise goals though.
I work with non-English Latin characters regularly and the UTF-8 allows working in Windows 7 with no issues. “WINDOWS-1252” has been useful for exporting into Microsoft readers such as Excel.

PS The Russian words were generated here http://generator.lorem-ipsum.info/_russian, so essentially meaningless.
PPS The warnings you mentioned remain still with no apparent important effects.

More Related Contents:

  1. RStudio not picking the encoding I’m telling it to use when reading a file
  2. How to source() .R file saved using UTF-8 encoding?
  3. how to use loop to subset 1000 csv files and do co-integration test in R?
  4. How to use Greek symbols in ggplot2?
  5. Read multiple CSV files into separate data frames
  6. How can you read a CSV file in R with different number of columns
  7. How to read in numbers with a comma as decimal separator?
  8. Subset of rows containing NA (missing) values in a chosen column of a data frame
  9. read.csv warning ‘EOF within quoted string’ prevents complete reading of file
  10. Prevent row names to be written to file when using write.csv
  11. how to read data in utf-8 format in R?
  12. Unicode Characters in ggplot2 PDF Output
  13. Read a UTF-8 text file with BOM
  14. ‘utf-8’ codec can’t decode byte 0xa0 in position 4276: invalid start byte
  15. Download a file from HTTPS using download.file()
  16. How to identify/delete non-UTF-8 characters in R
  17. Reading multiple csv files from a folder into a single dataframe in R [duplicate]
  18. Read and rbind multiple csv files
  19. Cannot read unicode .csv into R
  20. How to read csv file in R where some values contain the percent symbol (%)
  21. Is it possible to get the number of rows in a CSV file without opening it?
  22. How to append a whole dataframe to a CSV in R
  23. Using R to download zipped data file, extract, and import .csv
  24. Save imported csv data in vector – R
  25. Sequence length encoding using R
  26. R: How to save lists into csv?
  27. consolidating data frames in R
  28. Importing “csv” file with multiple-character separator to R?
  29. How can I merge many data frames from csv files when the ID column is implied?
  30. What does the error “arguments imply differing number of rows: x, y” mean?

Leave a Comment