Fastest way to write huge data in text file Java

by

You might try removing the BufferedWriter and just using the FileWriter directly. On a modern system there’s a good chance you’re just writing to the drive’s cache memory anyway.

It takes me in the range of 4-5 seconds to write 175MB (4 million strings) — this is on a dual-core 2.4GHz Dell running Windows XP with an 80GB, 7200-RPM Hitachi disk.

Can you isolate how much of the time is record retrieval and how much is file writing?

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.Writer;
import java.util.ArrayList;
import java.util.List;

public class FileWritingPerfTest {
    

private static final int ITERATIONS = 5;
private static final double MEG = (Math.pow(1024, 2));
private static final int RECORD_COUNT = 4000000;
private static final String RECORD = "Help I am trapped in a fortune cookie factory\n";
private static final int RECSIZE = RECORD.getBytes().length;

public static void main(String[] args) throws Exception {
    List<String> records = new ArrayList<String>(RECORD_COUNT);
    int size = 0;
    for (int i = 0; i < RECORD_COUNT; i++) {
        records.add(RECORD);
        size += RECSIZE;
    }
    System.out.println(records.size() + " 'records'");
    System.out.println(size / MEG + " MB");
    
    for (int i = 0; i < ITERATIONS; i++) {
        System.out.println("\nIteration " + i);
        
        writeRaw(records);
        writeBuffered(records, 8192);
        writeBuffered(records, (int) MEG);
        writeBuffered(records, 4 * (int) MEG);
    }
}

private static void writeRaw(List<String> records) throws IOException {
    File file = File.createTempFile("foo", ".txt");
    try {
        FileWriter writer = new FileWriter(file);
        System.out.print("Writing raw... ");
        write(records, writer);
    } finally {
        // comment this out if you want to inspect the files afterward
        file.delete();
    }
}

private static void writeBuffered(List<String> records, int bufSize) throws IOException {
    File file = File.createTempFile("foo", ".txt");
    try {
        FileWriter writer = new FileWriter(file);
        BufferedWriter bufferedWriter = new BufferedWriter(writer, bufSize);
    
        System.out.print("Writing buffered (buffer size: " + bufSize + ")... ");
        write(records, bufferedWriter);
    } finally {
        // comment this out if you want to inspect the files afterward
        file.delete();
    }
}

private static void write(List<String> records, Writer writer) throws IOException {
    long start = System.currentTimeMillis();
    for (String record: records) {
        writer.write(record);
    }
    // writer.flush(); // close() should take care of this
    writer.close(); 
    long end = System.currentTimeMillis();
    System.out.println((end - start) / 1000f + " seconds");
}
}

More Related Contents:

  1. File I/O Concept [closed]
  2. Create a file from a photo URI on Android
  3. How do I save a String to a text file using Java?
  4. Java IO implementation of unix/linux “tail -f”
  5. file.delete() returns false even though file.exists(), file.canRead(), file.canWrite(), file.canExecute() all return true
  6. In Java, how do I parse XML as a String instead of a file?
  7. Quickly read the last line of a text file?
  8. How do I sort very large files
  9. Recursively list all files within a directory using nio.file.DirectoryStream;
  10. Most efficient conversion of ResultSet to JSON?
  11. How to read a file from a jar file?
  12. Best Way to Write Bytes in the Middle of a File in Java
  13. Read a file line by line in reverse order
  14. Mocking Files in Java – Mock Contents – Mockito
  15. JDBC driver throws “ResultSet Closed” exception on empty ResultSet
  16. How to write and read java serialized objects into a file
  17. How to scan a folder in Java?
  18. Sort a file with huge volume of data given memory constraint
  19. How to convert InputStream to virtual File
  20. How to load and parse SVG documents
  21. Bulk insert in Java using prepared statements batch update
  22. Change file owner group under Linux with java.nio.Files
  23. How to append data to a file?
  24. Create folder in Android [duplicate]
  25. Accessing a file inside a .jar file [duplicate]
  26. How to execute JAVA program by double clicking on icon?
  27. deleting folder from java [duplicate]
  28. How to read from files with Files.lines(…).forEach(…)?
  29. Eclipse – Show generated class files
  30. Read directory inside JAR with InputStreamReader

Leave a Comment