‘utf-8’ codec can’t decode byte 0x92 in position 18: invalid start byte

by

The data is indeed not encoded as UTF-8; everything is ASCII except for that single 0x92 byte:

b'Korea, Dem. People\x92s Rep.'

Decode it as Windows codepage 1252 instead, where 0x92 is a fancy quote, ’:

df1 = pd.read_csv("https://raw.githubusercontent.com/tuyenhavan/Statistics/Dataset/World_Life_Expectancy.csv",
                  sep=";", encoding='cp1252')

Demo:

>>> import pandas as pd
>>> df1 = pd.read_csv("https://raw.githubusercontent.com/tuyenhavan/Statistics/Dataset/World_Life_Expectancy.csv",
...                   sep=";", encoding='cp1252')
>>> df1.head()
                   2000  2001  2002  2003  2004  2005  2006  2007  2008  2009  \
0     Afghanistan  55.1  55.5  55.9  56.2  56.6  57.0  57.4  57.8  58.2  58.6
1         Albania  74.3  74.7  75.2  75.5  75.8  76.1  76.3  76.5  76.7  76.8
2         Algeria  70.2  70.6  71.0  71.4  71.8  72.2  72.6  72.9  73.2  73.5
3  American Samoa    ..    ..    ..    ..    ..    ..    ..    ..    ..    ..
4         Andorra    ..    ..    ..    ..    ..    ..    ..    ..    ..    ..

   2010  2011  2012  2013  Unnamed: 15  2014  2015
0  59.0  59.3  59.7  60.0          NaN  60.4  60.7
1  77.0  77.2  77.4  77.6          NaN  77.8  78.0
2  73.8  74.1  74.3  74.6          NaN  74.8  75.0
3    ..    ..    ..    ..          NaN    ..    ..
4    ..    ..    ..    ..          NaN    ..    ..

I note however, that Pandas seems to take the HTTP headers at face value too and produces a Mojibake when you load your data from a URL. When I save the data directly to disk, then load it with pd.read_csv() the data is correctly decoded, but loading from the URL produces re-coded data:

>>> df1[' '][102]
'Korea, Dem. People’s Rep.'
>>> df1[' '][102].encode('cp1252').decode('utf8')
'Korea, Dem. People’s Rep.'

This is a known bug in Pandas. You can work around this by using urllib.request to load the URL and pass that to pd.read_csv() instead:

>>> import urllib.request
>>> with urllib.request.urlopen("https://raw.githubusercontent.com/tuyenhavan/Statistics/Dataset/World_Life_Expectancy.csv") as resp:
...     df1 = pd.read_csv(resp, sep=";", encoding='cp1252')
...
>>> df1[' '][102]
'Korea, Dem. People’s Rep.'

More Related Contents:

  1. Concatenate strings from several rows using Pandas groupby
  2. How to replace NaNs by preceding or next values in pandas DataFrame?
  3. Replace values in a pandas series via dictionary efficiently
  4. FutureWarning: elementwise comparison failed; returning scalar, but in the future will perform elementwise comparison
  5. Calculate average of every x rows in a table and create new table
  6. Moving Average Pandas
  7. How to deal with multi-level column names downloaded with yfinance
  8. Rolling Mean on pandas on a specific column
  9. Pandas deleting row with df.drop doesn’t work
  10. append dataframe to excel with pandas
  11. How to melt 2 columns at the same time?
  12. How can I see the formulas of an excel spreadsheet in pandas / python?
  13. How to remove or hide x-axis labels from a seaborn / matplotlib plot
  14. Unpack dictionary from Pandas Column
  15. What is the fastest and most efficient way to append rows to a DataFrame?
  16. Remove NaN ‘Cells’ without dropping the entire ROW (Pandas,Python3)
  17. Showing all index values when using multiIndexing in Pandas
  18. pandas.io.json.json_normalize with very nested json
  19. Number rows within group in increasing order in a pandas dataframe
  20. How to change the plot order of the categorical x-axis
  21. Updating a value in a pandas dataframe in an iterrows loop
  22. UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x8b in position 1: invalid start byte, while reading csv file in pandas
  23. append dictionary to data frame
  24. FutureWarning: The default value of regex will change from True to False in a future version
  25. Check if rows in one dataframe exist in another dataframe
  26. Pandas ParserError EOF character when reading multiple csv files to HDF5
  27. Python making combined bar and line plot with secondary y-axis
  28. AttributeError: module ‘numpy’ has no attribute ‘__version__’
  29. Pandas rank by multiple columns
  30. bash: Python import – Command not found for pandas

Leave a Comment