Detect text language in R

by

The textcat package does this. It can detect 74 ‘languages’ (more properly, language/encoding combinations), more with other extensions. Details and examples are in this freely available article:

Hornik, K., Mair, P., Rauch, J., Geiger, W., Buchta, C., & Feinerer, I. The textcat Package for n-Gram Based Text Categorization in R. Journal of Statistical Software, 52, 1-17.

Here’s the abstract:

Identifying the language used will typically be the first step in most
natural language processing tasks. Among the wide variety of language
identification methods discussed in the literature, the ones employing
the Cavnar and Trenkle (1994) approach to text categorization based on
character n-gram frequencies have been particularly successful. This
paper presents the R extension package textcat for n-gram based text
categorization which implements both the Cavnar and Trenkle approach
as well as a reduced n-gram approach designed to remove redundancies
of the original approach. A multi-lingual corpus obtained from the
Wikipedia pages available on a selection of topics is used to
illustrate the functionality of the package and the performance of the
provided language identification methods.

And here’s one of their examples:

library("textcat")
textcat(c(
  "This is an English sentence.",
  "Das ist ein deutscher Satz.",
  "Esta es una frase en espa~nol."))
[1] "english" "german" "spanish"

More Related Contents:

  1. How can I find compound words, removing spaces between them and replace them in my corpus?
  2. R-Project no applicable method for ‘meta’ applied to an object of class “character”
  3. Finding 2 & 3 word Phrases Using R TM Package
  4. R tm package invalid input in ‘utf8towcs’
  5. bigrams instead of single words in termdocument matrix using R and Rweka
  6. Recognize PDF table using R
  7. Emoticons in Twitter Sentiment Analysis in r
  8. list of word frequencies using R
  9. Text-mining with the tm-package – word stemming
  10. R text file and text mining…how to load data
  11. How do I combine two data-frames based on two columns? [duplicate]
  12. Finding percentage in a sub-group using group_by and summarise
  13. Add a prefix to column names
  14. ggplot2 heatmap with colors for ranged values
  15. How to extract the row with min or max values?
  16. ggplot2 plot without axes, legends, etc
  17. How to get google search results
  18. How to install a package from a download zip file
  19. Read and rbind multiple csv files
  20. Loading many files at once?
  21. R – Group by variable and then assign a unique ID [duplicate]
  22. How I can select rows from a dataframe that do not match?
  23. Remove facet_wrap labels completely
  24. data.table row-wise sum, mean, min, max like dplyr?
  25. Triple exclamation marks on R
  26. R command line passing a filename to script in arguments (Windows)
  27. Calculate percentage of each category in each group in R [duplicate]
  28. How to specify columns in facet_grid OR how to change labels in facet_wrap
  29. R optimization with equality and inequality constraints
  30. R – ggplot line color (using geom_line) doesn’t change

Leave a Comment