Python 3 – Can pickle handle byte objects larger than 4GB?

by

Here is a simple workaround for issue 24658. Use pickle.loads or pickle.dumps and break the bytes object into chunks of size 2**31 - 1 to get it in or out of the file.

import pickle
import os.path

file_path = "pkl.pkl"
n_bytes = 2**31
max_bytes = 2**31 - 1
data = bytearray(n_bytes)

## write
bytes_out = pickle.dumps(data)
with open(file_path, 'wb') as f_out:
    for idx in range(0, len(bytes_out), max_bytes):
        f_out.write(bytes_out[idx:idx+max_bytes])

## read
bytes_in = bytearray(0)
input_size = os.path.getsize(file_path)
with open(file_path, 'rb') as f_in:
    for _ in range(0, input_size, max_bytes):
        bytes_in += f_in.read(max_bytes)
data2 = pickle.loads(bytes_in)

assert(data == data2)

More Related Contents:

  1. Saving and loading multiple objects in pickle file?
  2. Using pickle.dump – TypeError: must be str, not bytes
  3. Unpickling a python 2 object with python 3
  4. Pickle incompatibility of numpy arrays between Python 2 and 3
  5. How to select random key with common values?
  6. Integer division in Python 2 and Python 3
  7. Matplotlib: Specify format of floats for tick labels
  8. How to set Python’s default version to 3.x on OS X? [duplicate]
  9. Play an Animated GIF in python with tkinter
  10. “list index out of range” when using sys.argv[1]
  11. Py2exe for Python 3.0
  12. Why can’t I use the method __cmp__ in Python 3 as for Python 2?
  13. Python Binomial Coefficient
  14. Why doesn’t Python recognize my utf-8 encoded source file?
  15. rounding errors in Python floor division
  16. How do I include files with pyinstaller?
  17. Accessing items in an collections.OrderedDict by index
  18. How can I see the formulas of an excel spreadsheet in pandas / python?
  19. Python 3 – Encode/Decode vs Bytes/Str [duplicate]
  20. Serialising an Enum member to JSON
  21. How to explain the str.maketrans function in Python 3.6?
  22. Why is a class __dict__ a mappingproxy?
  23. What is the difference between int() and floor() in Python 3?
  24. Changing User Agent in Python 3 for urrlib.request.urlopen
  25. How to fix “no module named ‘app_one'”
  26. Calling Python script from C++ and using its output
  27. Format a number with commas to separate thousands
  28. AttributeError: Could not find PyAudio; check installation…can’t use speech Recognition
  29. Run process with realtime output to a Tkinter GUI
  30. how to add a coroutine to a running asyncio loop?

Leave a Comment