PowerShell script runs when pasted into the PowerShell window, but not when run from shortcut

by

To clarify:

  • It is perfectly fine to use Unicode (non-ASCII-range) quotation marks such as “ in PowerShell – see the bottom section.

  • However, in order to use such characters in script files, these files must use a Unicode character encoding such as UTF-8 or UTF-16LE (“Unicode”).

Your problem was that your script file was saved as UTF-8 without a BOM, which causes Windows PowerShell (but not PowerShell Core) to misinterpret it, because it defaults to “ANSI” encoding, i.e., the single-byte legacy encoding associated with the legacy system locale a.k.a language for non-Unicode programs (e.g., Windows-1252 in the US and Western Europe), which PowerShell calls Default.

While replacing the Unicode quotation marks with their ASCII counterparts solves the immediate problem, any other non-ASCII-range characters in the script would continue to be misinterpreted.

  • The proper solution is to re-save the file as UTF-8 with BOM.
  • It is a good habit to form to routinely save all PowerShell scripts (source code) as UTF-8 with BOM, because that ensures that they’re interpreted the same, irrespective of any given computer’s system locale.

To demonstrate the specific problem:

  • “, the LEFT DOUBLE QUOTATION MARK (U+201C) Unicode character, is encoded as three bytes in UTF-8 format: 0xE2 0x80 0x9C.

  • You can verify this via the output from '“' | Format-Hex -Encoding Utf8 (only the byte sequence matters here; the printed characters. on the right are not representative in this case).

  • When Windows PowerShell reads this sequence as “ANSI”-encoded, it considers each byte a character in its own right, which is why you saw three characters for the single “ in your output, namely “.

  • You can verify this with [Text.Encoding]::Default.GetString([byte[]] (0xE2, 0x80, 0x9C)) (from PowerShell Core, use [Text.Encoding]::GetEncoding([cultureinfo]::CurrentCulture.TextInfo.ANSICodePage).GetString([byte[]] (0xE2, 0x80, 0x9C))).

Interchangeable use of ASCII-range and Unicode quotation marks, dashes, and whitespace in PowerShell:

In a properly encoded input file, PowerShell allows interchangeable use of the following quotation and punctuation characters; e.g., "hi", ”hi” and even "hi„ are equivalent.

Note:

  • Important: The above describes interchangeable syntactic use of these characters; if you use such characters in identifiers (which you shouldn’t) or in strings[1], they are not treated the same.

  • The above was in part gleaned from the source code on GitHub (class SpecialCharacters in file parserutils.cs).

[1] There are limited exceptions: given that PowerShell’s -eq operator compares string using the invariant culture rather than performing ordinal comparison, space-character variations may be treated the same in string comparisons, depending on the host platform; e.g., "foo bar" -eq "foo`u{a0}bar" yields $true on macOS and Linux (but not Windows!), because the regular ASCII-range space is considered equal to the no-break space (U+00A0) there.

More Related Contents:

  1. Character-encoding problem with string literal in source code
  2. Displaying Unicode in Powershell
  3. Changing PowerShell’s default output encoding to UTF-8
  4. UTF-8 output from PowerShell
  5. Trouble with UTF-8 characters; what I see is not what I stored
  6. What’s the difference between UTF-8 and UTF-8 without BOM?
  7. UTF8 to/from wide char conversion in STL
  8. What is the difference between UTF-8 and Unicode?
  9. Convert Unicode to ASCII without errors in Python
  10. What is the proper way to URL encode Unicode characters?
  11. FPDF utf-8 encoding (HOW-TO)
  12. How do you properly use WideCharToMultiByte
  13. How does UTF-8 “variable-width encoding” work?
  14. Read a UTF-8 text file with BOM
  15. Powershell string variable with UTF-8 encoding
  16. How do I correct the character encoding of a file?
  17. C programming: How to program for Unicode?
  18. What’s the difference between UTF-8 and UTF-8 with BOM?
  19. How do I call the PowerShell CLI robustly, with respect to character encoding, input and output streams, quoting and escaping?
  20. Powershell ConvertFrom-Json Encoding Special Characters Issue
  21. How do I encode Unicode character codes in a PowerShell string literal?
  22. ASCII vs Unicode + UTF-8
  23. Inheriting and overriding functions of a std::string?
  24. Bytes in a unicode Python string
  25. Is there a way to make a PowerShell script work by double clicking a .ps1 file?
  26. Scheduled Task Powershell Script – Runs OK as user account, but not as SYSTEM
  27. Powershell color formatting with format operator
  28. How to make an authenticated web request in Powershell?
  29. Unicode in PDF
  30. Uploading file to HTTP via Powershell

Leave a Comment