Best method of saving data

by

If your data are pretty simple, like just collections of collections of strings or numbers, I would use json. What JSON is, is a string representation of simple data types and combinations of simple data types. Once you use the json module to convert your data to a string, you write it to a file yourself.

It’s super simple:

>>> my_data = [range(5) for i in range(5)]
>>> my_data
[[0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4]]
>>> import json
>>> json.dumps(my_data)
'[[0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4]]'

Then just write that string to a file. When you want to reload it, like so:

>>> import json
>>> string_from_file
'[[0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4]]'
>>> my_saved_data = json.loads(string_from_file)
>>> my_saved_data
[[0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4], [0, 1, 2, 3, 4]]

If your data are more complicated, and involves classes other than the built-in collection objects, pickle is a better choice. One very important thing to know about pickle is that there are security vulnerabilities in pickle, and it’s a bad idea to unpickle anything you yourself didn’t pickle. pickle is vulnerable to the security problems detailed in this article: http://www.kalzumeus.com/2013/01/31/what-the-rails-security-issue-means-for-your-startup/

If the size of your data is very large, or you will be saving/loading it frequently, or for any reason using json and saving to a local file is inadequate, then a database will be the way to go.

More Related Contents:

  1. Saving an Object (Data persistence)
  2. How to make a class JSON serializable
  3. How to get string objects instead of Unicode from JSON?
  4. How to JSON serialize sets?
  5. Best way to save a trained model in PyTorch? [closed]
  6. Basic http file downloading and saving to disk in python?
  7. Storing Python dictionaries
  8. How I can I lazily read multiple JSON values from a file/stream in Python?
  9. In Python, how can you load YAML mappings as OrderedDicts?
  10. How do I save a trained model in PyTorch?
  11. is not JSON serializable
  12. Convert a python dict to a string and back
  13. django-rest-framework 3.0 create or update in nested serializer
  14. How can I save an image with PIL?
  15. Saving the highscore for a game?
  16. Serializing binary data in Python
  17. Flask jsonify a list of objects
  18. Django REST: Uploading and serializing multiple images
  19. When to use Serializer’s create() and ModelViewset’s perform_create()
  20. Serialising an Enum member to JSON
  21. save numpy array in append mode
  22. List of objects to JSON with Python
  23. TypeError: Object of type ‘bytes’ is not JSON serializable
  24. Setting the fmt option in numpy.savetxt
  25. Serialize the @property methods in a Python class
  26. Common use-cases for pickle in Python
  27. How to save & load xgboost model? [closed]
  28. Serializing list to JSON [duplicate]
  29. Django: When to customize save vs using post-save signal
  30. What difference between pickle and _pickle in python 3?

Leave a Comment