How to use numpy.genfromtxt when first column is string and the remaining columns are numbers?

by

By default, np.genfromtxt uses dtype=float: that’s why you string columns are converted to NaNs because, after all, they’re Not A Number…

You can ask np.genfromtxt to try to guess the actual type of your columns by using dtype=None:

>>> from StringIO import StringIO
>>> test = "a,1,2\nb,3,4"
>>> a = np.genfromtxt(StringIO(test), delimiter=",", dtype=None)
>>> print a
array([('a',1,2),('b',3,4)], dtype=[('f0', '|S1'),('f1', '<i8'),('f2', '<i8')])

You can access the columns by using their name, like a['f0']…

Using dtype=None is a good trick if you don’t know what your columns should be. If you already know what type they should have, you can give an explicit dtype. For example, in our test, we know that the first column is a string, the second an int, and we want the third to be a float. We would then use

>>> np.genfromtxt(StringIO(test), delimiter=",", dtype=("|S10", int, float))
array([('a', 1, 2.0), ('b', 3, 4.0)], 
      dtype=[('f0', '|S10'), ('f1', '<i8'), ('f2', '<f8')])

Using an explicit dtype is much more efficient than using dtype=None and is the recommended way.

In both cases (dtype=None or explicit, non-homogeneous dtype), you end up with a structured array.

[Note: With dtype=None, the input is parsed a second time and the type of each column is updated to match the larger type possible: first we try a bool, then an int, then a float, then a complex, then we keep a string if all else fails. The implementation is rather clunky, actually. There had been some attempts to make the type guessing more efficient (using regexp), but nothing that stuck so far]

More Related Contents:

  1. Replacing values represented by 'UN' with NaN
  2. ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
  3. pandas select from Dataframe using startswith
  4. What’s the fastest way in Python to calculate cosine similarity given sparse matrix data?
  5. Force NumPy ndarray to take ownership of its memory in Cython
  6. How to normalize a NumPy array to within a certain range?
  7. numpy np.apply_along_axis function speed up?
  8. how to extract frequency associated with fft values in python
  9. How do I use np.newaxis?
  10. Python insert numpy array into sqlite3 database
  11. Error: Microsoft Visual C++ 10.0 is required (Unable to find vcvarsall.bat) when running Python script [duplicate]
  12. python numpy machine epsilon
  13. Deprecation status of the NumPy matrix class
  14. How to apply numpy.linalg.norm to each row of a matrix?
  15. Get week start date (Monday) from a date column in Python (pandas)?
  16. What is dtype(‘O’), in pandas?
  17. Python baseline correction library
  18. numpy float: 10x slower than builtin in arithmetic operations?
  19. What is the purpose of meshgrid in Python / NumPy?
  20. Index 2D numpy array by a 2D array of indices without loops
  21. Python: find contour lines from matplotlib.pyplot.contour()
  22. No speedup when summing uint16 vs uint64 arrays with NumPy?
  23. Partition array into N chunks with Numpy
  24. Compute a confidence interval from sample data assuming unknown distribution
  25. 3-dimensional array in numpy
  26. What is the most efficient way to check if a value exists in a NumPy array?
  27. ImportError: DLL load failed when importing Numpy installed in conda virtual environment
  28. Fast Way to slice image into overlapping patches and merge patches to image
  29. How to smooth a curve for a dataset
  30. How can I make a numpy ndarray from bytes?

Leave a Comment