How can I treat command-line arguments as UTF-8 in Perl?

by

Outside data sources are tricky in Perl. For command-line arguments, you’re probably getting them as the encoding specified in your locale. Don’t rely on your locale to be the same as someone else who might run your program.

You have to find out what that is then convert to Perl’s internal format. Fortunately, it’s not that hard.

The I18N::Langinfo module has the stuff you need to get the encoding:

    use I18N::Langinfo qw(langinfo CODESET);
    my $codeset = langinfo(CODESET);

Once you know the encoding, you can decode them to Perl strings:

    use Encode qw(decode);
    @ARGV = map { decode $codeset, $_ } @ARGV;

Although Perl encodes internal strings as UTF-8, you shouldn’t ever think or know about that. You just decode whatever you get, which turns it into Perl’s internal representation for you. Trust that Perl will handle everything else. When you need to store the data, ensure that you use the encoding you like.

If you know that your setup is UTF-8 and the terminal will give you the command-line arguments as UTF-8, you can use the A option with Perl’s -C switch. This tells your program to assume the arguments are encoded as UTF-8:

% perl -CA program

More Related Contents:

  1. Why does modern Perl avoid UTF-8 by default?
  2. How can I output UTF-8 from Perl?
  3. Use of ‘use utf8;’ gives me ‘Wide character in print’
  4. Why do my Perl tests fail with use encoding ‘utf8’?
  5. How can I guess the encoding of a string in Perl?
  6. How to retrieve the values of required keys in Perl [closed]
  7. Perl: Meaning of double squared brackets after a string?
  8. What’s the difference between UTF-8 and UTF-8 without BOM?
  9. Saving utf-8 texts with json.dumps as UTF8, not as \u escape sequence
  10. u’\ufeff’ in Python string
  11. Why is three-argument open calls with autovivified filehandles a Perl best practice?
  12. What’s the difference between Unicode and UTF-8? [duplicate]
  13. How do I check if a string is unicode or ascii?
  14. Manually converting unicode codepoints into UTF-8 and UTF-16
  15. How can I parse dates and convert time zones in Perl?
  16. Which one is good practice, a lexical filehandle or a typeglob?
  17. How many characters can be mapped with Unicode?
  18. Perl memory usage profiling and leak detection?
  19. How should I use the “my” keyword in Perl?
  20. Global symbol requires explicit package name
  21. Python DictWriter writing UTF-8 encoded CSV files
  22. How do I include functions from another file in my Perl script?
  23. How to make python 3 print() utf8
  24. What reasons are there to prefer glob over readdir (or vice-versa) in Perl?
  25. Do C++11 regular expressions work with UTF-8 strings?
  26. Is There a Way to Match Any Unicode Alphabetic Character?
  27. How can I find the version of an installed Perl module?
  28. How can I escape an arbitrary string for use as a command line argument in Bash?
  29. Nested subroutines and Scoping in Perl
  30. Why do hash keys have different order when printing?

Leave a Comment