best way to preserve numpy arrays on disk

by

I’ve compared performance (space and time) for a number of ways to store numpy arrays. Few of them support multiple arrays per file, but perhaps it’s useful anyway.

benchmark for numpy array storage

Npy and binary files are both really fast and small for dense data. If the data is sparse or very structured, you might want to use npz with compression, which’ll save a lot of space but cost some load time.

If portability is an issue, binary is better than npy. If human readability is important, then you’ll have to sacrifice a lot of performance, but it can be achieved fairly well using csv (which is also very portable of course).

More details and the code are available at the github repo.

More Related Contents:

  1. Preserve custom attributes when pickling subclass of numpy array
  2. ImportError: No module named ‘pandas.indexes’
  3. Python pickle protocol choice?
  4. How to hash a large object (dataset) in Python?
  5. ValueError: setting an array element with a sequence
  6. Numpy `logical_or` for more than two arguments
  7. Concatenate a NumPy array to another NumPy array
  8. ImportError: numpy.core.multiarray failed to import
  9. How do you use the ellipsis slicing syntax in Python?
  10. How to check if a value is in the list in selection from pandas data frame?
  11. How to add items into a numpy array
  12. multithreaded blas in python/numpy
  13. shuffling/permutating a DataFrame in pandas
  14. SQL-like window functions in PANDAS: Row Numbering in Python Pandas Dataframe
  15. numpy array assignment problem
  16. What techniques can be used to measure performance of pandas/numpy solutions
  17. Python Numpy TypeError: ufunc ‘isfinite’ not supported for the input types
  18. Find index where elements change value numpy
  19. Why is np.where faster than pd.apply
  20. 2d convolution using python and numpy
  21. Converting a 2D numpy array to a structured array
  22. Numpy difference between neighboring elements
  23. new pythonic style for shared axes square subplots in matplotlib?
  24. How to resample a dataframe with different functions applied to each column?
  25. Converting 2D Numpy array of grayscale values to a PIL image
  26. Convert a column of timestamps into periods in pandas
  27. Consecutive, Overlapping Subsets of Array (NumPy, Python)
  28. A numpy array unexpectedly changes when changing another one despite being separate
  29. Generate random array of floats between a range
  30. Understanding scipy deconvolve

Leave a Comment