Java Read Large Text File With 70million line of text

by

1) I am sure there is no difference speedwise, both use FileInputStream internally and buffering

2) You can take measurements and see for yourself

3) Though there’s no performance benefits I like the 1.7 approach

try (BufferedReader br = Files.newBufferedReader(Paths.get("test.txt"), StandardCharsets.UTF_8)) {
    for (String line = null; (line = br.readLine()) != null;) {
        //
    }
}

4) Scanner based version

    try (Scanner sc = new Scanner(new File("test.txt"), "UTF-8")) {
        while (sc.hasNextLine()) {
            String line = sc.nextLine();
        }
        // note that Scanner suppresses exceptions
        if (sc.ioException() != null) {
            throw sc.ioException();
        }
    }

5) This may be faster than the rest

try (SeekableByteChannel ch = Files.newByteChannel(Paths.get("test.txt"))) {
    ByteBuffer bb = ByteBuffer.allocateDirect(1000);
    for(;;) {
        StringBuilder line = new StringBuilder();
        int n = ch.read(bb);
        // add chars to line
        // ...
    }
}

it requires a bit of coding but it can be really faster because of ByteBuffer.allocateDirect. It allows OS to read bytes from file to ByteBuffer directly, without copying

6) Parallel processing would definitely increase speed. Make a big byte buffer, run several tasks that read bytes from file into that buffer in parallel, when ready find first end of line, make a String, find next…

More Related Contents:

  1. Process a file line by line
  2. Scanner is skipping nextLine() after using next() or nextFoo()?
  3. How to append text to an existing file in Java?
  4. Scanner is skipping nextLine() after using next() or nextFoo()?
  5. How do I create a Java string from the contents of a file?
  6. How can I read a large text file line by line using Java?
  7. How do I get the file extension of a file in Java?
  8. byte[] to file in Java
  9. How to simulate keyboard presses in java?
  10. In Java is it possible to re-open System.in after closing it
  11. What throws an IOException in Java?
  12. How to copy file from one location to another location?
  13. How to use readline() method in Java?
  14. Java: Reading integers from a file into an array
  15. FileInputStream vs FileReader
  16. File exists and IS directory, but listFiles() returns null
  17. Modify the content of a file using Java
  18. How to make a folder hidden using java
  19. Get file name from FileOutputStream
  20. What is InputStream & Output Stream? Why and when do we use them?
  21. Listing files in a directory matching a pattern in Java [duplicate]
  22. In Java, what is the default location for newly created files?
  23. java.io.StreamCorruptedException: invalid stream header: 7371007E
  24. Given a Java InputStream, how can I determine the current offset in the stream?
  25. How to break a file into pieces using Java?
  26. How can I read comma separated values from a text file in Java?
  27. difference between flush and close function in case of filewriter in java
  28. Resetting Standard output Stream
  29. java file input with rewind()/reset() capability
  30. Will using longs instead of ints benefit in 64bit java

Leave a Comment