Python CGI – UTF-8 doesn’t work

by

For CGI, using print() requires that the correct codec has been set up for output. print() writes to sys.stdout and sys.stdout has been opened with a specific encoding and how that is determined is platform dependent and can differ based on how the script is run. Running your script as a CGI script means you pretty much do not know what encoding will be used.

In your case, the web server has set the locale for text output to a fixed encoding other than UTF-8. Python uses that locale setting to produce output in in that encoding, and without the <meta> header your browser correctly guesses that encoding (or the server has communicated it in the Content-Type header), but with the <meta> header you are telling it to use a different encoding, one that is incorrect for the data produced.

You can write directly to sys.stdout.buffer, after explicitly encoding to UTF-8. Make a helper function to make this easier:

import sys

def enc_print(string='', encoding='utf8'):
    sys.stdout.buffer.write(string.encode(encoding) + b'\n')

enc_print("Content-type:text/html")
enc_print()
enc_print("""
    <!doctype html>
    <html>
    <head>
        <meta charset="UTF-8">
    </head>
    <body>
        şöğıçü
    </body>
    </html>
""")

Another approach is to replace sys.stdout with a new io.TextIOWrapper() object that uses the codec you need:

import sys
import io

def set_output_encoding(codec, errors="strict"):
    sys.stdout = io.TextIOWrapper(
        sys.stdout.detach(), errors=errors,
        line_buffering=sys.stdout.line_buffering)

set_output_encoding('utf8')

print("Content-type:text/html")
print()
print("""
    <!doctype html>
    <html>
    <head></head>
    <body>
        şöğıçü
    </body>
    </html>
""")

More Related Contents:

  1. error UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0xff in position 0: invalid start byte
  2. How do I unescape HTML entities in a string in Python 3.1? [duplicate]
  3. Beautiful Soup findAll doesn’t find them all
  4. printing UTF-8 in Python 3 using Sublime Text 3
  5. Why doesn’t Python recognize my utf-8 encoded source file?
  6. Python 3 print() function with Farsi/Arabic characters [duplicate]
  7. Should I use encoding declaration in Python 3?
  8. python 3 open() default encoding
  9. How to display uploaded image in HTML page using FastAPI & Jinja2?
  10. What encoding does open() use by default?
  11. Changing the “locale preferred encoding” in Python 3 in Windows
  12. Posting html form values to python script
  13. UnicodeDecodeError in Python when reading a file, how to ignore the error and jump to the next line?
  14. Python 3 CSV file giving UnicodeDecodeError: ‘utf-8’ codec can’t decode byte error when I print
  15. Set encoding in Python 3 CGI scripts
  16. Python random.choice same output
  17. Python: read object from file in python
  18. How to deal with certificates using Selenium?
  19. PyQt4 Wait in thread for user input from GUI
  20. Upload Image using POST form data in Python-requests
  21. importing izip from itertools module gives NameError in Python 3.x
  22. Python 3.4.0 with MySQL database
  23. pd.read_html() imports a list rather than a dataframe
  24. How to embed image or picture in jupyter notebook, either from a local machine or from a web resource?
  25. How to change the plot order of the categorical x-axis
  26. How to zoom in and out of an image pygame and using the mousposition as the center of the zoom
  27. String-based enum in Python
  28. Django Admin: Using a custom widget for only one model field
  29. How do I change the volume of the sound or music in PyGame?
  30. What difference between pickle and _pickle in python 3?

Leave a Comment