Read .csv file from URL into Python 3.x – _csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)

by

The problem relies on urllib returning bytes. As a proof, you can try to download the csv file with your browser and opening it as a regular file and the problem is gone.

A similar problem was addressed here.

It can be solved decoding bytes to strings with the appropriate encoding. For example:

import csv
import urllib.request

url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv"
ftpstream = urllib.request.urlopen(url)
csvfile = csv.reader(ftpstream.read().decode('utf-8'))  # with the appropriate encoding 
data = [row for row in csvfile]

The last line could also be: data = list(csvfile) which can be easier to read.

By the way, since the csv file is very big, it can slow and memory-consuming. Maybe it would be preferable to use a generator.

EDIT:
Using codecs as proposed by Steven Rumbalski so it’s not necessary to read the whole file to decode. Memory consumption reduced and speed increased.

import csv
import urllib.request
import codecs

url = "ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/file_list.csv"
ftpstream = urllib.request.urlopen(url)
csvfile = csv.reader(codecs.iterdecode(ftpstream, 'utf-8'))
for line in csvfile:
    print(line)  # do something with line

Note that the list is not created either for the same reason.

More Related Contents:

  1. A simple maths operation on a CSV column using python [closed]
  2. How to add a new column to a CSV file?
  3. TypeError: a bytes-like object is required, not ‘str’ in python and CSV
  4. csv.Error: iterator should return strings, not bytes
  5. portable way to write csv file in python 2 or python 3
  6. python pandas read_csv delimiter in column data
  7. pandas.read_csv from string or package data
  8. UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x8b in position 1: invalid start byte, while reading csv file in pandas
  9. Pandas ParserError EOF character when reading multiple csv files to HDF5
  10. How to get the previous URL from a POST in Django
  11. Python 3 CSV file giving UnicodeDecodeError: ‘utf-8’ codec can’t decode byte error when I print
  12. How to write List of lists in csv file in python [duplicate]
  13. Python3: writing csv files
  14. Parsing a pipe-delimited file in Python
  15. Print elements of list horizontally [closed]
  16. Get raw key value
  17. Python import csv to list
  18. Download file from web in Python 3
  19. How to make separator in pandas read_csv more flexible wrt whitespace, for irregular separators?
  20. Python, TypeError: unhashable type: ‘list’
  21. How to convert ‘binary string’ to normal string in Python3?
  22. How to redirect with post data (Django)
  23. ImportError: No module named ‘django.core.urlresolvers’
  24. Recursively iterate through all subdirectories using pathlib
  25. How to transpose a dataset in a csv file?
  26. How to make a custom object iterable?
  27. Identifier normalization: Why is the micro sign converted into the Greek letter mu?
  28. How to create virtual env with python3
  29. asterisk in tuple, list and set definitions, double asterisk in dict definition
  30. Python sort list by first number in a string

Leave a Comment