How can I serialize a numpy array while preserving matrix dimensions?

by

pickle.dumps or numpy.save encode all the information needed to reconstruct an arbitrary NumPy array, even in the presence of endianness issues, non-contiguous arrays, or weird structured dtypes. Endianness issues are probably the most important; you don’t want array([1]) to suddenly become array([16777216]) because you loaded your array on a big-endian machine. pickle is probably the more convenient option, though save has its own benefits, given in the npy format rationale.

I’m giving options for serializing to JSON or a bytestring, because the original questioner needed JSON-serializable output, but most people coming here probably don’t.

The pickle way:

import pickle
a = # some NumPy array

# Bytestring option
serialized = pickle.dumps(a)
deserialized_a = pickle.loads(serialized)

# JSON option
# latin-1 maps byte n to unicode code point n
serialized_as_json = json.dumps(pickle.dumps(a).decode('latin-1'))
deserialized_from_json = pickle.loads(json.loads(serialized_as_json).encode('latin-1'))

numpy.save uses a binary format, and it needs to write to a file, but you can get around that with io.BytesIO:

a = # any NumPy array
memfile = io.BytesIO()
numpy.save(memfile, a)

serialized = memfile.getvalue()
serialized_as_json = json.dumps(serialized.decode('latin-1'))
# latin-1 maps byte n to unicode code point n

And to deserialize:

memfile = io.BytesIO()

# If you're deserializing from a bytestring:
memfile.write(serialized)
# Or if you're deserializing from JSON:
# memfile.write(json.loads(serialized_as_json).encode('latin-1'))
memfile.seek(0)
a = numpy.load(memfile)

More Related Contents:

  1. NumPy array is not JSON serializable
  2. Convert numpy type to python
  3. Json Encoder AND Decoder for complex numpy arrays
  4. Convert a 3D array to RGB valued array [closed]
  5. Why am I seeing “TypeError: string indices must be integers”?
  6. Is there a memory efficient and fast way to load big JSON files?
  7. Replacing Numpy elements if condition is met
  8. Matplotlib log scale tick label number formatting
  9. What does the term “broadcasting” mean in Pandas documentation?
  10. String to Dictionary in Python [duplicate]
  11. Save / load scipy sparse csr_matrix in portable data format
  12. Working with TIFFs (import, export) in Python using numpy
  13. How to load all entries in an infinite scroll at once to parse the HTML in python
  14. Shift elements in a numpy array
  15. Numpy and line intersections
  16. Numpy: Creating a complex array from 2 real ones?
  17. Are Decimal ‘dtypes’ available in NumPy?
  18. Fill zero values of 1d numpy array with last non-zero values
  19. Error “Import Error: No module named numpy” on Windows
  20. Binary numpy array to list of integers?
  21. Weighted percentile using numpy
  22. test for membership in a 2d numpy array
  23. OpenAI GPT-3 API error: “Cannot specify both model and engine”
  24. Find large number of consecutive values fulfilling condition in a numpy array
  25. How to automatically annotate maximum value in pyplot
  26. Why does python/numpy’s += mutate the original array?
  27. Multiple matrix multiplication
  28. Is there a MATLAB accumarray equivalent in numpy?
  29. Generate random locations within a triangular domain
  30. sparse 3d matrix/array in Python?

Leave a Comment