Is there a memory efficient and fast way to load big JSON files?

by

There was a duplicate to this question that had a better answer. See https://stackoverflow.com/a/10382359/1623645, which suggests ijson.

Update:

I tried it out, and ijson is to JSON what SAX is to XML. For instance, you can do this:

import ijson
for prefix, the_type, value in ijson.parse(open(json_file_name)):
    print prefix, the_type, value

where prefix is a dot-separated index in the JSON tree (what happens if your key names have dots in them? I guess that would be bad for Javascript, too…), theType describes a SAX-like event, one of 'null', 'boolean', 'number', 'string', 'map_key', 'start_map', 'end_map', 'start_array', 'end_array', and value is the value of the object or None if the_type is an event like starting/ending a map/array.

The project has some docstrings, but not enough global documentation. I had to dig into ijson/common.py to find what I was looking for.

More Related Contents:

  1. Can I get JSON to load into an OrderedDict?
  2. How to extract multiple JSON objects from one file?
  3. Converting dictionary to JSON
  4. json.dumps vs flask.jsonify
  5. Python/Json:Expecting property name enclosed in double quotes
  6. Python read JSON file and modify
  7. How to get JSON from webpage into Python script
  8. Python’s json module, converts int dictionary keys to strings
  9. python: read lines from compressed text files
  10. Django REST Framework POST nested objects
  11. Serializing a Python namedtuple to json
  12. Python: Trying to Deserialize Multiple JSON objects in a file with each object spanning multiple but consistently spaced number of lines
  13. Cast Flask form value to int
  14. Check if key exists and iterate the JSON array using Python
  15. Dump to JSON adds additional double quotes and escaping of quotes
  16. How to return data in JSON format using FastAPI?
  17. Python JSON TypeError list indices must be integers or slices, not str
  18. How to send JSON as part of multipart POST-request
  19. How to create a FastAPI endpoint that can accept either Form or JSON body?
  20. Python JSON encoder to support datetime?
  21. json reading error json.decoder.JSONDecodeError: Invalid \escape
  22. Converting Flask form data to JSON only gets first value
  23. How to make JSON flattening memory efficient?
  24. Reading an JSON file from S3 using Python boto3
  25. Serializing list to JSON [duplicate]
  26. json.dump throwing “TypeError: {…} is not JSON serializable” on seemingly valid object?
  27. ValueError when using pandas.read_json
  28. Create a Pandas DataFrame from deeply nested JSON
  29. SQLAlchemy: filtering on values stored in nested list of the JSONB field
  30. Python 2.6 JSON decoding performance

Leave a Comment