Python: Trying to Deserialize Multiple JSON objects in a file with each object spanning multiple but consistently spaced number of lines

by

Load 6 extra lines instead, and pass the string to json.loads():

with open(file) as f:
    for line in f:
        # slice the next 6 lines from the iterable, as a list.
        lines = [line] + list(itertools.islice(f, 6))
        jfile = json.loads(''.join(lines))

        # do something with jfile

json.load() will slurp up more than just the next object in the file, and islice(f, 0, 7) would read only the first 7 lines, rather than read the file in 7-line blocks.

You can wrap reading a file in blocks of size N in a generator:

from itertools import islice, chain

def lines_per_n(f, n):
    for line in f:
        yield ''.join(chain([line], itertools.islice(f, n - 1)))

then use that to chunk up your input file:

with open(file) as f:
    for chunk in lines_per_n(f, 7):
        jfile = json.loads(chunk)

        # do something with jfile

Alternatively, if your blocks turn out to be of variable length, read until you have something that parses:

with open(file) as f:
    for line in f:
        while True:
            try:
                jfile = json.loads(line)
                break
            except ValueError:
                # Not yet a complete JSON value
                line += next(f)

        # do something with jfile

More Related Contents:

  1. Check if a dictionary contain data true of false, then send a response back
  2. Why am I seeing “TypeError: string indices must be integers”?
  3. Convert JSON string to dict using Python
  4. Is there a memory efficient and fast way to load big JSON files?
  5. Python: json.loads returns items prefixing with ‘u’
  6. Convert Pandas Dataframe to nested JSON
  7. How to convert an XML string to a dictionary?
  8. How to flatten a nested JSON recursively, with flatten_json
  9. Let JSON object accept bytes or let urlopen output strings
  10. String to Dictionary in Python [duplicate]
  11. Python – How to convert JSON File to Dataframe
  12. How to compare two JSON objects with the same elements in a different order equal?
  13. How to load all entries in an infinite scroll at once to parse the HTML in python
  14. Convert a python dict to a string and back
  15. Python json.loads fails with `ValueError: Invalid control character at: line 1 column 33 (char 33)`
  16. django-rest-framework 3.0 create or update in nested serializer
  17. How do I send a POST request as a JSON?
  18. Flask jsonify a list of objects
  19. Updates to JSON field don’t persist to DB
  20. Convert a HTML Table to JSON
  21. python: read json and loop dictionary
  22. pandas.io.json.json_normalize with very nested json
  23. How to convert a list of numbers to jsonarray in Python
  24. Write two-dimensional list to JSON file
  25. How do I check if a string is valid JSON in Python?
  26. How to change json encoding behaviour for serializable python object?
  27. Python json.loads changes the order of the object
  28. Insert multiple documents in Elasticsearch – bulk doc formatter
  29. Convert JSON to SQLite in Python – How to map json keys to database columns properly?
  30. How to convert csv to json in python?

Leave a Comment