How to detect the right encoding for read.csv?

by

First of all based on more general question on StackOverflow it is not possible to detect encoding of file in 100% certainty.

I’ve struggle this many times and come to non-automatic solution:

Use iconvlist to get all possible encodings:

codepages <- setNames(iconvlist(), iconvlist())

Then read data using each of them

x <- lapply(codepages, function(enc) try(read.table("encoding.asc",
                   fileEncoding=enc,
                   nrows=3, header=TRUE, sep="\t"))) # you get lots of errors/warning here

Important here is to know structure of file (separator, headers). Set encoding using fileEncoding argument. Read only few rows.
Now you could lookup on results:

unique(do.call(rbind, sapply(x, dim)))
#        [,1] [,2]
# 437       14    2
# CP1200     3   29
# CP12000    0    1

Seems like correct one is that with 3 rows and 29 columns, so lets see them:

maybe_ok <- sapply(x, function(x) isTRUE(all.equal(dim(x), c(3,29))))
codepages[maybe_ok]
#    CP1200    UCS-2LE     UTF-16   UTF-16LE      UTF16    UTF16LE 
#  "CP1200"  "UCS-2LE"   "UTF-16" "UTF-16LE"    "UTF16"  "UTF16LE"

You could look on data too

x[maybe_ok]

For your file all this encodings returns identical data (partially because there is some redundancy as you see).

If you don’t know specific of your file you need to use readLines with some changes in workflow (e.g. you can’t use fileEncoding, must use length instead of dim, do more magic to find correct ones).

More Related Contents:

  1. Specify custom Date format for colClasses argument in read.table/read.csv
  2. ‘Incomplete final line’ warning when trying to read a .csv file into R
  3. Invalid multibyte string in read.csv
  4. Force character vector encoding from “unknown” to “UTF-8” in R
  5. How to read only lines that fulfil a condition from a csv into R?
  6. Specifying colClasses in the read.csv
  7. Why am I getting X. in my column names when reading a data frame?
  8. Read a UTF-8 text file with BOM
  9. read.csv doesn’t seem to detect factors in R 4.0.0
  10. How to check if CSV file has a comma or a semicolon as separator?
  11. read.csv, header on first line, skip second line [duplicate]
  12. Combine several data frames in the global environment by row (rbind)
  13. Imported a csv-dataset to R but the values becomes factors
  14. How can I read the header but also skip lines – read.table()?
  15. Manipulating files with non-English names in R
  16. Weird error in R when importing (64-bit) integer with many digits
  17. geom_rect and alpha – does this work with hard coded values?
  18. Using loops with knitr to produce multiple pdf reports… need a little help to get me over the hump
  19. Argument is of length zero in if statement
  20. How do I name the “row names” column in r
  21. How to disable “Save workspace image?” prompt in R?
  22. How to use tidyr::separate when the number of needed variables is unknown [duplicate]
  23. finding close match from data frame 1 in data fame 2
  24. Showing string in formula and not as variable in lm fit
  25. What does the @ symbol mean in R?
  26. Select along one of n dimensions in array
  27. Calculate number of days between two dates in r
  28. Data input via shinyTable in R shiny application
  29. Deleting every n-th row in a dataframe
  30. Calculate monthly average of ts object

Leave a Comment