How do I use the ‘json’ module to read in one JSON object at a time?

by

Generally speaking, putting more than one JSON object into a file makes that file invalid, broken JSON. That said, you can still parse data in chunks using the JSONDecoder.raw_decode() method.

The following will yield complete objects as the parser finds them:

from json import JSONDecoder
from functools import partial


def json_parse(fileobj, decoder=JSONDecoder(), buffersize=2048):
    buffer=""
    for chunk in iter(partial(fileobj.read, buffersize), ''):
         buffer += chunk
         while buffer:
             try:
                 result, index = decoder.raw_decode(buffer)
                 yield result
                 buffer = buffer[index:].lstrip()
             except ValueError:
                 # Not enough data to decode, read more
                 break

This function will read chunks from the given file object in buffersize chunks, and have the decoder object parse whole JSON objects from the buffer. Each parsed object is yielded to the caller.

Use it like this:

with open('yourfilename', 'r') as infh:
    for data in json_parse(infh):
        # process object

Use this only if your JSON objects are written to a file back-to-back, with no newlines in between. If you do have newlines, and each JSON object is limited to a single line, you have a JSON Lines document, in which case you can use Loading and parsing a JSON file with multiple JSON objects in Python instead.

More Related Contents:

  1. Check if a dictionary contain data true of false, then send a response back
  2. Why am I seeing “TypeError: string indices must be integers”?
  3. Convert JSON string to dict using Python
  4. Is there a memory efficient and fast way to load big JSON files?
  5. Python: json.loads returns items prefixing with ‘u’
  6. How to convert an XML string to a dictionary?
  7. Python json parser allow duplicate keys
  8. How to extract a single value from JSON response?
  9. How can I access the nested data in this complex JSON, which includes another JSON document as one of the strings?
  10. Loading initial data with Django 1.7 and data migrations
  11. Accessing JSON elements
  12. Python load json file with UTF-8 BOM header
  13. Loading JSONL file as JSON objects
  14. Why do I get an error “‘unicode’ object does not support item deletion” when trying to delete values from a JSON object?
  15. Convert a HTML Table to JSON
  16. python: read json and loop dictionary
  17. pandas.io.json.json_normalize with very nested json
  18. How to convert a list of numbers to jsonarray in Python
  19. Write two-dimensional list to JSON file
  20. How do I check if a string is valid JSON in Python?
  21. How to change json encoding behaviour for serializable python object?
  22. Python json.loads changes the order of the object
  23. Insert multiple documents in Elasticsearch – bulk doc formatter
  24. What is the difference between json.dumps and json.load? [closed]
  25. How to merge two json string in Python?
  26. json.decoder.JSONDecodeError: Extra data: line 2 column 1 (char 190) [duplicate]
  27. Issue with merging multiple JSON files in Python
  28. How to Fix JSON Key Values without double-quotes?
  29. Handle JSON Decode Error when nothing returned
  30. Convert nested JSON to CSV file in Python

Leave a Comment