how to parse a huge JSON file without loading it in memory

You should definitely check different approaches and libraries. If you are really take care about performance check: Gson, Jackson and JsonPath libraries to do that and choose the fastest one. Definitely you have to load the whole JSON file on local disk, probably TMP folder and parse it after that.

Simple JsonPath solution could look like below:

import com.jayway.jsonpath.DocumentContext;
import com.jayway.jsonpath.JsonPath;

import java.io.File;

public class JsonPathApp {
    public static void main(String[] args) throws Exception {
        File jsonFile = new File("./resource/test.json").getAbsoluteFile();

        DocumentContext documentContext = JsonPath.parse(jsonFile);
        System.out.println("" + documentContext.read("$.a"));
        System.out.println("" + documentContext.read("$.b"));
        System.out.println("" + documentContext.read("$.d"));
    }
}

Notice, that I do not create any POJO, just read given values using JSONPath feature similarly to XPath. The same you can do with Jackson:

import com.fasterxml.jackson.databind.JsonNode;
import com.fasterxml.jackson.databind.ObjectMapper;

import java.io.File;

public class JsonPathApp {
    public static void main(String[] args) throws Exception {
        File jsonFile = new File("./resource/test.json").getAbsoluteFile();

        ObjectMapper mapper = new ObjectMapper();
        JsonNode root = mapper.readTree(jsonFile);
        System.out.println(root.get("a"));
        System.out.println(root.get("b"));
        System.out.println(root.get("d"));
    }
}

We do not need JSONPath because values we need are directly in root node. As you can see, API looks almost the same. We can also create POJO structure:

import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.databind.ObjectMapper;

import java.io.File;
import java.math.BigDecimal;

public class JsonPathApp {
    public static void main(String[] args) throws Exception {
        File jsonFile = new File("./resource/test.json").getAbsoluteFile();

        ObjectMapper mapper = new ObjectMapper();
        Pojo pojo = mapper.readValue(jsonFile, Pojo.class);
        System.out.println(pojo);
    }
}

@JsonIgnoreProperties(ignoreUnknown = true)
class Pojo {
    private Integer a;
    private BigDecimal b;
    private Integer d;

    // getters, setters
}

Even so, both libraries allow to read JSON payload directly from URL I suggest to download it in another step using best approach you can find. For more info, read this article: Download a File From an URL in Java.

Leave a Comment