Is there a workaround for Java’s poor performance on walking huge directories?

by

Although it’s not pretty, I solved this kind of problem once by piping the output of dir/ls to a file before starting my app, and passing in the filename.

If you needed to do it within the app, you could just use system.exec(), but it would create some nastiness.

You asked. The first form is going to be blazingly fast, the second should be pretty fast as well.

Be sure to do the one item per line (bare, no decoration, no graphics), full path and recurse options of your selected command.

EDIT:

30 minutes just to get a directory listing, wow.

It just struck me that if you use exec(), you can get it’s stdout redirected into a pipe instead of writing it to a file.

If you did that, you should start getting the files immediately and be able to begin processing before the command has completed.

The interaction may actually slow things down, but maybe not–you might give it a try.

Wow, I just went to find the syntax of the .exec command for you and came across this, possibly exactly what you want (it lists a directory using exec and “ls” and pipes the result into your program for processing): good link in wayback (Jörg provided in a comment to replace this one from sun that Oracle broke)

Anyway, the idea is straightforward but getting the code right is annoying. I’ll go steal some codes from the internets and hack them up–brb


/**
 * Note: Only use this as a last resort!  It's specific to windows and even
 * at that it's not a good solution, but it should be fast.
 * 
 * to use it, extend FileProcessor and call processFiles("...") with a list
 * of options if you want them like /s... I highly recommend /b
 * 
 * override processFile and it will be called once for each line of output.
 */
import java.io.*;

public abstract class FileProcessor
{
   public void processFiles(String dirOptions)
   {
      Process theProcess = null;
      BufferedReader inStream = null;

      // call the Hello class
      try
      {
          theProcess = Runtime.getRuntime().exec("cmd /c dir " + dirOptions);
      }
      catch(IOException e)
      {
         System.err.println("Error on exec() method");
         e.printStackTrace();  
      }

      // read from the called program's standard output stream
      try
      {
         inStream = new BufferedReader(
                                new InputStreamReader( theProcess.getInputStream() ));  
         processFile(inStream.readLine());
      }
      catch(IOException e)
      {
         System.err.println("Error on inStream.readLine()");
         e.printStackTrace();  
      }

   } // end method
   /** Override this method--it will be called once for each file */
   public abstract void processFile(String filename);


} // end class

And thank you code donor at IBM

More Related Contents:

  1. When to use StringBuilder in Java [duplicate]
  2. The performance impact of using instanceof in Java
  3. Does setting Java objects to null do anything anymore?
  4. Performance of Java matrix math libraries? [closed]
  5. Fastest way to iterate over all the chars in a String
  6. How do I speed up the gwt compiler?
  7. Performance of StringTokenizer class vs. String.split method in Java
  8. Is there any performance reason to declare method parameters final in Java?
  9. Why is System.out.println so slow?
  10. Android RecyclerView Scrolling Performance
  11. Java HashMap performance optimization / alternative
  12. Does variable = null set it for garbage collection
  13. Fastest way to iterate an Array in Java: loop variable vs enhanced for statement [duplicate]
  14. JTable Calls Custom Cell Renderer Method… Continuously
  15. Fastest way to sum integers in text file
  16. Java try/catch performance, is it recommended to keep what is inside the try clause to a minimum?
  17. Most efficient way of converting String to Integer in java
  18. fastest (low latency) method for Inter Process Communication between Java and C/C++
  19. What is the best macro-benchmarking tool / framework to measure a single-threaded complex algorithm in Java? [closed]
  20. How to speed up Java VM (JVM) startup time?
  21. compareTo with primitives -> Integer / int
  22. Writing image to servlet response with best performance
  23. Java Collections and Garbage Collector
  24. Which “if” construct is faster – statement or ternary operator?
  25. Java’s varargs performance
  26. Fastest way of reading relatively huge byte-files in Java
  27. Java TCP socket: data transfer is slow
  28. Java compile speed vs Scala compile speed
  29. Is there a performance difference between Javac debug on and off?
  30. Big execution time difference between java Lambda vs Anonymous class

Leave a Comment