Using jq how can I split a very large JSON file into multiple files, each a specific quantity of objects?

by

[EDIT: This answer has been revised in accordance with the revision to the question.]

The key to using jq to solve the problem is the -c command-line option, which produces output in JSON-Lines format (i.e., in the present case, one object per line). You can then use a tool such as awk or split to distribute those lines amongst several files.

If the file is not too big, then the simplest would be to start the pipeline with:

jq -c '.[]' INPUTFILE

If the file is too big to fit comfortably in memory, then you could use jq’s streaming parser, like so:

jq -cn --stream 'fromstream(1|truncate_stream(inputs))'

For further discussion about the streaming parser, see e.g. the relevant section in the jq FAQ: https://github.com/stedolan/jq/wiki/FAQ#streaming-json-parser

Partitioning

For different approaches to partitioning the output produced in the first step, see for example How to split a large text file into smaller files with equal number of lines?

If it is required that each of the output files be an array of objects, then I’d probably use awk to perform both the partitioning and the re-constitution in one step, but there are many other reasonable approaches.

If the input is a sequence of JSON objects

For reference, if the original file consists of a stream or sequence of JSON objects, then the appropriate invocation would be:

jq -n -c inputs INPUTFILE

Using inputs in this manner allows arbitrarily many objects to be processed efficiently.

More Related Contents:

  1. Passing bash variable to jq
  2. How to convert arbitrary simple JSON to CSV using jq?
  3. How to merge 2 JSON objects from 2 files using jq?
  4. Select objects based on value of variable in object using jq
  5. jq not working on tag name with dashes and numbers
  6. Iterating through JSON array in Shell script
  7. passing arguments to jq filter
  8. Modify a key-value in a json using jq in-place
  9. how to parse a JSON String with jq (or other alternatives)?
  10. How to convert a JSON object to key=value format in jq?
  11. jq: Object cannot be csv-formatted, only array
  12. jq: print key and value for each entry in an object
  13. How to format a JSON string as a table using jq?
  14. Get outputs from jq on a single line
  15. How do I update a single value in a json document using jq?
  16. Exporting JSON to environment variables
  17. How to convert a JSON object stream into an array with jq
  18. Convert a JSON array to a bash array of strings
  19. How to use jq when the variable has reserved characters?
  20. How to filter array of objects by element property values using jq?
  21. How to get key names from JSON using jq
  22. Reusable function to group_by but return an object with group as key
  23. Split a JSON file into separate files
  24. Create JSON using jq from pipe-separated keys and values in bash
  25. Constructing a JSON object from a bash associative array
  26. How do I modify fields inside the new PostgreSQL JSON datatype?
  27. How to get Date object from json Response in typescript
  28. Convert MongoDB BsonDocument to valid JSON in C#
  29. Read Json file into a data.frame without nested lists
  30. Reading JSON with Apache Spark – `corrupt_record`

Leave a Comment