Character reading from file in Python

by

Ref: http://docs.python.org/howto/unicode

Reading Unicode from a file is therefore simple:

import codecs
with codecs.open('unicode.rst', encoding='utf-8') as f:
    for line in f:
        print repr(line)

It’s also possible to open files in update mode, allowing both reading and writing:

with codecs.open('test', encoding='utf-8', mode="w+") as f:
    f.write(u'\u4500 blah blah blah\n')
    f.seek(0)
    print repr(f.readline()[:1])

EDIT: I’m assuming that your intended goal is just to be able to read the file properly into a string in Python. If you’re trying to convert to an ASCII string from Unicode, then there’s really no direct way to do so, since the Unicode characters won’t necessarily exist in ASCII.

If you’re trying to convert to an ASCII string, try one of the following:

  1. Replace the specific unicode chars with ASCII equivalents, if you are only looking to handle a few special cases such as this particular example

  2. Use the unicodedata module’s normalize() and the string.encode() method to convert as best you can to the next closest ASCII equivalent (Ref https://web.archive.org/web/20090228203858/http://techxplorer.com/2006/07/18/converting-unicode-to-ascii-using-python): 

    >>> teststr
u'I don\xe2\x80\x98t like this'
>>> unicodedata.normalize('NFKD', teststr).encode('ascii', 'ignore')
'I donat like this'

More Related Contents:

  1. Replace non-ASCII characters with a single space
  2. Why does Python print unicode characters when the default encoding is ASCII?
  3. UnicodeEncodeError: ‘ascii’ codec can’t encode character at special name [duplicate]
  4. Python – Unicode to ASCII conversion
  5. How can I remove non-ASCII characters but leave periods and spaces?
  6. How to check if a string in Python is in ASCII?
  7. Convert Unicode to ASCII without errors in Python
  8. Python string prints as [u’String’]
  9. Windows cmd encoding change causes Python crash
  10. Python UnicodeDecodeError – Am I misunderstanding encode?
  11. Python Unicode Encode Error
  12. Get a list of all the encodings Python can encode to
  13. Where is Python’s “best ASCII for this Unicode” database? [closed]
  14. Non-ASCII characters in Matplotlib
  15. How do I check if a string is unicode or ascii?
  16. How to print UTF-8 encoded text to the console in Python < 3?
  17. ‘UCS-2’ codec can’t encode characters in position 1050-1050
  18. Why doesn’t Python recognize my utf-8 encoded source file?
  19. In Python, how do I decode GZIP encoding?
  20. Python 3: How to specify stdin encoding
  21. Python ascii utf unicode
  22. How do I treat an ASCII string as unicode and unescape the escaped characters in it in python?
  23. ElementTree and unicode
  24. How to detect string byte encoding?
  25. Convert “\x” escaped string into readable string in python
  26. python encoding utf-8
  27. How does unicodedata.normalize(form, unistr) work?
  28. Python – dealing with mixed-encoding files
  29. UnicodeDecodeError when performing os.walk
  30. Printing a string prints ‘u’ before the string in Python?

Leave a Comment