Powershell string variable with UTF-8 encoding

by

For a comprehensive overview of how PowerShell interacts with external programs, which includes sending data to them, see this answer.

When PowerShell interprets output from external programs (such as ytd in your case), it assumes that the output uses the character encoding reflected in [Console]::OutputEncoding.

Note:

  • Interpreting refers to cases where PowerShell captures (e.g., $output = ...), relays (e.g., ... | Select-String ...), or redirects (e.g., ... > output.txt) the external program’s output.

  • By contrast, printing directly to the display may not be affected, because PowerShell then isn’t involved, and certain CLIs adjust their behavior when their stdout isn’t redirected to print directly to the console with full Unicode support (which explains why the characters looked as expected in your console when ytd‘s output printed directly to it).

If the encoding reported by [Console]::OutputEncoding is not the same encoding used by the external program at hand, PowerShell misinterprets the output.

To fix that, you must (temporarily) set [Console]::OutputEncoding] to match the encoding used by the external program.

For instance, let’s assume an executable foo.exe that outputs UTF-8-encoded text:

# Save the current encoding and switch to UTF-8.
$prev = [Console]::OutputEncoding
[Console]::OutputEncoding = [System.Text.UTF8Encoding]::new()

# PowerShell now interprets foo's output correctly as UTF-8-encoded.
# and $output will correctly contain CJK characters.
$output = foo https://example.org -e

# Restore the previous encoding.
[Console]::OutputEncoding = $prev

Important:

  • [Console]::OutputEncoding by default reflects the encoding associated with the legacy system locale’s OEM code page, as reported by chcp (e.g. 437 on US-English systems).

    • Recent versions of Windows 10 now allow setting the system locale to code page 65001 (UTF-8) (the feature is still in beta as of Window 10 version 1909), which is great, considering that most modern command-line utilities “speak” UTF-8 – but note that making this system-wide change has far-reaching consequences – see this answer.

With the specific program at hand, youtube-dl, js2010 has discovered that capturing in a variable works without extra effort if you pass --encoding utf-16.

The reason this works is that the resulting UTF16-LE-encoded output is preceded by a BOM (Byte-Order Mark).

(Note that --encoding utf-8 does not work, because youtube-dl then does not emit a BOM.)

Windows PowerShell is capable of detecting and properly decoding UTF-16LE-encoded and UTF-8-encoded text irrespective of the effective [Console]::OutputEncoding] IF AND ONLY IF the output is preceded by a BOM.

Caveats:

  • This does not work in PowerShell Core (v6+, on any of the supported platforms).

  • Even in Windows PowerShell you’ll rarely be able to take advantage of this obscure behavior, because using a BOM in stdout output is atypical (it is typically only used in files).

More Related Contents:

  1. Changing PowerShell’s default output encoding to UTF-8
  2. Character-encoding problem with string literal in source code
  3. PowerShell script runs when pasted into the PowerShell window, but not when run from shortcut
  4. UTF-8 output from PowerShell
  5. Pin program to taskbar using PS in Windows 10
  6. How do I call the PowerShell CLI robustly, with respect to character encoding, input and output streams, quoting and escaping?
  7. Powershell ConvertFrom-Json Encoding Special Characters Issue
  8. Powershell script to delete 15 days old .txt files [duplicate]
  9. How to run an EXE file in PowerShell with parameters with spaces and quotes
  10. Use PowerShell to run virus scan on multiple servers
  11. Unable to access UNC Paths in Powershell remote session
  12. Fastest Way to get a uniquely index item from the property of an array
  13. PowerShell IE9 ComObject has all null properties after navigating to webpage
  14. save PSCredential in the file
  15. PowerShell’s Invoke-RestMethod equivalent of curl -u (Basic Authentication)
  16. How do I run a PowerShell script when the computer starts?
  17. ConvertTo-Csv Output without quotes
  18. PowerShell – get process ID of called application
  19. Start-Process with PowerShell.exe exhibits different behavior with embedded single quotes and double quotes
  20. Can the following Nested foreach loop be simplified in PowerShell?
  21. Add Write-Progress to Get-Job/Wait-Job
  22. Powershell with exchange– How do I append all verbose output to a file
  23. How to handle command-line arguments in PowerShell
  24. How to send email with PowerShell
  25. powershell function switch param with string array
  26. Check if a file exists or not in Windows PowerShell?
  27. Variable scoping in PowerShell
  28. PowerShell launch script in new instance
  29. Read a Csv file with powershell and capture corresponding data
  30. How to return specific data type from PowerShell function i.e. DataTable

Leave a Comment