How can I guess the encoding of a string in Perl?

by

To find out in which encoding something unknown uses, you just have to try and look. The modules Encode::Detect and Encode::Guess automate that. (If you have trouble compiling Encode::Detect, try its fork Encode::Detective instead.)

use Encode::Detect::Detector;
my $unknown = "\x{54}\x{68}\x{69}\x{73}\x{20}\x{79}\x{65}\x{61}\x{72}\x{20}".
              "\x{49}\x{20}\x{77}\x{65}\x{6e}\x{74}\x{20}\x{74}\x{6f}\x{20}".
              "\x{b1}\x{b1}\x{be}\x{a9}\x{20}\x{50}\x{65}\x{72}\x{6c}\x{20}".
              "\x{77}\x{6f}\x{72}\x{6b}\x{73}\x{68}\x{6f}\x{70}\x{2e}";
my $encoding_name = Encode::Detect::Detector::detect($unknown);
print $encoding_name; # gb18030

use Encode;
my $string = decode($encoding_name, $unknown);

I find encode 'ascii' is a lame solution for getting rid of non-ASCII characters. Everything will be substituted with questions marks; this is too lossy to be useful. 

# Bad example; don't do this.
use utf8;
use Encode;
my $string = 'This year I went to 北京 Perl workshop.';
print encode('ascii', $string); # This year I went to ?? Perl workshop.

If you want readable ASCII text, I recommend Text::Unidecode instead. This, too, is a lossy encoding, but not as terrible as plain encode above.

use utf8;
use Text::Unidecode;
my $string = 'This year I went to 北京 Perl workshop.';
print unidecode($string); # This year I went to Bei Jing  Perl workshop.

However, avoid those lossy encodings if you can help it. In case you want to reverse the operation later, pick either one of PERLQQ or XMLCREF.

use utf8;
use Encode qw(encode PERLQQ XMLCREF);
my $string = 'This year I went to 北京 Perl workshop.';
print encode('ascii', $string, PERLQQ);  # This year I went to \x{5317}\x{4eac} Perl workshop.
print encode('ascii', $string, XMLCREF); # This year I went to 北京 Perl workshop.

More Related Contents:

  1. Why does modern Perl avoid UTF-8 by default?
  2. How can I output UTF-8 from Perl?
  3. What is the best way to slurp a file into a string in Perl?
  4. What’s the difference between single and double quotes in Perl?
  5. Use of ‘use utf8;’ gives me ‘Wide character in print’
  6. How can I treat command-line arguments as UTF-8 in Perl?
  7. Failing to install MIME::Base64 using cpan [closed]
  8. Declaring a 2D coordinate in Perl / C / C++ [closed]
  9. Why can’t Perl find my file that is in ClearCase?
  10. In Perl, how can I read an entire file into a string?
  11. What’s the difference between a hash and hash reference in Perl?
  12. Multiline search replace with Perl
  13. How do I “use” a Perl module in a directory not in @INC?
  14. How can I process options using Perl in -n or -p mode?
  15. How can I dynamically include Perl modules without using eval?
  16. Can a valid Unicode string contain FFFF? Is Java/CharacterIterator broken?
  17. How can I compile my Perl script so it can be executed on systems without ‘perl’ installed?
  18. Why does my Perl script exit with 137?
  19. How to reverse a Unicode string
  20. Convert between string, u16string & u32string
  21. Perl read line by line
  22. What is the difference between Perl’s ( or, and ) and ( ||, && ) short-circuit operators?
  23. How do I change, delete, or insert a line in a file, or append to the beginning of a file in Perl?
  24. How can I install a specific version of a set of Perl modules?
  25. Obtain a switch/case behaviour in Perl 5
  26. How do I extract the domain out of an URL?
  27. Why does $foo->{bar} autovivify but %$foo doesn’t?
  28. What is internal representation of string in Python 3.x
  29. How do I retrieve the terminal width in Perl?
  30. converting narrow string to wide string

Leave a Comment