JSON generally isn’t very good for this sort of incremental use; there’s no standard way to serialise multiple objects so that they can easily be loaded one at a time, without parsing the whole lot.
The object per line solution that you’re using is seen elsewhere too. Scrapy calls it ‘JSON lines’:
- https://docs.scrapy.org/en/latest/topics/exporters.html?highlight=exporters#jsonitemexporter
- http://www.enricozini.org/2011/tips/python-stream-json/
You can do it slightly more Pythonically:
for jsonline in f:
yield json.loads(jsonline) # or do the processing in this loop
I think this is about the best way – it doesn’t rely on any third party libraries, and it’s easy to understand what’s going on. I’ve used it in some of my own code as well.