Read large files in Java

by

To save memory, do not unnecessarily store/duplicate the data in memory (i.e. do not assign them to variables outside the loop). Just process the output immediately as soon as the input comes in.

It really doesn’t matter whether you’re using BufferedReader or not. It will not cost significantly much more memory as some implicitly seem to suggest. It will at highest only hit a few % from performance. The same applies on using NIO. It will only improve scalability, not memory use. It will only become interesting when you’ve hundreds of threads running on the same file.

Just loop through the file, write every line immediately to other file as you read in, count the lines and if it reaches 100, then switch to next file, etcetera.

Kickoff example:

String encoding = "UTF-8";
int maxlines = 100;
BufferedReader reader = null;
BufferedWriter writer = null;

try {
    reader = new BufferedReader(new InputStreamReader(new FileInputStream("/bigfile.txt"), encoding));
    int count = 0;
    for (String line; (line = reader.readLine()) != null;) {
        if (count++ % maxlines == 0) {
            close(writer);
            writer = new BufferedWriter(new OutputStreamWriter(new FileOutputStream("/smallfile" + (count / maxlines) + ".txt"), encoding));
        }
        writer.write(line);
        writer.newLine();
    }
} finally {
    close(writer);
    close(reader);
}

More Related Contents:

  1. File I/O Concept [closed]
  2. How to read a file and put it into an array of objects?
  3. How do I save a String to a text file using Java?
  4. Java IO implementation of unix/linux “tail -f”
  5. How to create a temporary directory/folder in Java?
  6. file.delete() returns false even though file.exists(), file.canRead(), file.canWrite(), file.canExecute() all return true
  7. Quickly read the last line of a text file?
  8. How to set the maximum memory usage for JVM?
  9. How do I get the file name from a String containing the Absolute file path?
  10. Determine if two files store the same content
  11. How do I use Java to read from a file that is actively being written to?
  12. How to read a file from a jar file?
  13. Access resource files in Android
  14. Does variable = null set it for garbage collection
  15. Accessing files with spaces in filename from Java
  16. What are some Java memory management best practices? [closed]
  17. What is the best way to generate a unique and short file name in Java
  18. How to send byte[] as pdf to browser in java web application?
  19. adding progress bar to each table cell for file progress – Java
  20. java read file from network device
  21. read complete file without using loop in java
  22. How to execute JAVA program by double clicking on icon?
  23. Java – Search for files in a directory
  24. Where does the JVM store primitive variables?
  25. Parsing very large XML documents (and a bit more) in java
  26. How to create JList with icon and text?
  27. Determining binary/text file type in Java?
  28. Java: how to get a File from an escaped URL?
  29. How to read a verbose:GC output?
  30. Reading from a text file and storing in a String [duplicate]

Leave a Comment