Loading UTF-8 file in Python 3 using numpy.genfromtxt

by

Note the original 2015 date. Since then genfromtxt has gotten an encoding parameter.

In Python3 I can do:

In [224]: txt = "Côte d'Ivoire"
In [225]: x = np.zeros((2,),dtype="U20")
In [226]: x[0] = txt
In [227]: x
Out[227]: 
array(["Côte d'Ivoire", ''],   dtype="<U20")

Which means I probably could open a ‘UTF-8’ file (regular, not byte mode), and readlines, and assign them to elements of an array like x.

But genfromtxt insists on operating with byte strings (ascii) which can’t handle the larger UTF-8 set (7 bytes v 8). So I need to apply decode at some point to get an U array.

I can load it into a ‘S’ array with genfromtxt:

In [258]: txt="Côte d'Ivoire"
In [259]: a=np.genfromtxt([txt.encode()],delimiter=",",dtype="S20")
In [260]: a
Out[260]: 
array(b"C\xc3\xb4te d'Ivoire",  dtype="|S20")

and apply decode to individual elements:

In [261]: print(a.item().decode())
Côte d'Ivoire

In [325]: print _
Côte d'Ivoire

Or use np.char.decode to apply it to each element of an array:

In [263]: np.char.decode(a)
Out[263]: 
array("Côte d'Ivoire", dtype="<U13")
In [264]: print(_)
Côte d'Ivoire

genfromtxt lets me specify converters:

In [297]: np.genfromtxt([txt.encode()],delimiter=",",dtype="U20",
    converters={0:lambda x: x.decode()})
Out[297]: 
array("Côte d'Ivoire", dtype="<U20")

If the csv has a mix of strings and numbers, this converters approach will be easier to use than the np.char.decode. Just specify the converter for each string column.

(See my earlier edits for Python2 tries).

More Related Contents:

  1. Dump a NumPy array into a csv file
  2. Reading a UTF8 CSV file with Python
  3. Python out of memory on large CSV file (numpy)
  4. General Unicode/UTF-8 support for csv files in Python 2.6
  5. selecting across multiple columns with python pandas?
  6. How to write UTF-8 in a CSV file
  7. Using numpy.genfromtxt to read a csv file with strings containing commas
  8. ‘utf-8’ codec can’t decode byte 0xa0 in position 4276: invalid start byte
  9. Precision lost while using read_csv in pandas
  10. Python DictWriter writing UTF-8 encoded CSV files
  11. Numpy loading csv TOO slow compared to Matlab
  12. load csv into 2D matrix with numpy for plotting
  13. pandas.read_csv from string or package data
  14. module ‘pandas’ has no attribute ‘read_csv
  15. Extract csv file specific columns to list in Python
  16. Python 3 CSV file giving UnicodeDecodeError: ‘utf-8’ codec can’t decode byte error when I print
  17. Load CSV to Pandas MultiIndex DataFrame
  18. Plot NetworkX Graph from Adjacency Matrix in CSV file
  19. ValueError: x and y must be the same size
  20. Pandas df.to_csv(“file.csv” encode=”utf-8″) still gives trash characters for minus sign
  21. Bin values based on ranges with pandas [duplicate]
  22. How can I convert JSON to CSV?
  23. String concatenation of two pandas columns
  24. Python & MySql: Unicode and Encoding
  25. binning data in python with scipy/numpy
  26. How do I count the occurrence of a certain item in an ndarray?
  27. Calculating Pearson correlation and significance in Python
  28. How to split a numpy array in fixed size chunks with and without overlap?
  29. Why does numpy.linalg.solve() offer more precise matrix inversions than numpy.linalg.inv()?
  30. Pandas in AWS lambda gives numpy error

Leave a Comment