UnicodeDecodeError when performing os.walk

by

Right I just spent some time sorting through this error, and wordier answers here aren’t getting at the underlying issue:

The problem is, if you pass a unicode string into os.walk(), then os.walk starts getting unicode back from os.listdir() and tries to keep it as ASCII (hence ‘ascii’ decode error). When it hits a unicode only special character which str() can’t translate, it throws the exception.

The solution is to force the starting path you pass to os.walk to be a regular string – i.e. os.walk(str(somepath)). This means os.listdir returns regular byte-like strings and everything works the way it should.

You can reproduce this problem (and show it’s solution works) trivially like:

  1. Go into bash in some directory and run touch $(echo -e "\x8b\x8bThis is a bad filename") which will make some test files.

  2. Now run the following Python code (iPython Qt is handy for this) in the same directory:

    l = []
for root,dir,filenames in os.walk(unicode('.')):
    l.extend([ os.path.join(root, f) for f in filenames ])
print l

And you’ll get a UnicodeDecodeError.

  1. Now try running:

    l = []
for root,dir,filenames in os.walk('.'):
    l.extend([ os.path.join(root, f) for f in filenames ])
print l

No error and you get a print out!

Thus the safe way in Python 2.x is to make sure you only pass raw text to os.walk(). You absolutely should not pass unicode or things which might be unicode to it, because os.walk will then choke when an internal ascii conversion fails.

More Related Contents:

  1. How do I check if a string is unicode or ascii?
  2. Why doesn’t Python recognize my utf-8 encoded source file?
  3. Python ascii utf unicode
  4. ElementTree and unicode
  5. Convert UTF-16 to UTF-8 and remove BOM?
  6. Convert “\x” escaped string into readable string in python
  7. python encoding utf-8
  8. Python – dealing with mixed-encoding files
  9. Changing default encoding of Python?
  10. Unicode (UTF-8) reading and writing to files in Python
  11. Replace non-ASCII characters with a single space
  12. What are Unicode, UTF-8, and UTF-16?
  13. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xef in position 1
  14. Convert Unicode to ASCII without errors in Python
  15. Character reading from file in Python
  16. Python & MySql: Unicode and Encoding
  17. Python and BeautifulSoup encoding issues [duplicate]
  18. Dangers of sys.setdefaultencoding(‘utf-8’)
  19. How to convert a string to utf-8 in Python
  20. Writing UTF-8 String to MySQL with Python
  21. General Unicode/UTF-8 support for csv files in Python 2.6
  22. MySQL “incorrect string value” error when save unicode string in Django
  23. How to print UTF-8 encoded text to the console in Python < 3?
  24. UnicodeEncodeError: ‘ascii’ codec can’t encode character at special name [duplicate]
  25. How to correctly parse UTF-8 encoded HTML to Unicode strings with BeautifulSoup? [duplicate]
  26. Why declare unicode by string in python?
  27. ‘utf-8’ codec can’t decode byte 0xa0 in position 4276: invalid start byte
  28. UnicodeDecodeError: ‘ascii’ codec can’t decode byte 0xc3 in position 23: ordinal not in range(128)
  29. How to detect string byte encoding?
  30. SQLite, python, unicode, and non-utf data

Leave a Comment