How to retrieve a list of directories QUICKLY in Java?

by

As has already been mentioned, this is basicly a hardware problem. Disk access is always slow, and most file systems aren’t really designed to handle directories with that many files.

If you for some reason have to store all the files in the same directory, I think you’ll have to maintain your own cache. This could be done using a local database such as sqlite, HeidiSQL or HSQL. If you want extreme performance, use a java TreeSet and cache it in memory. This means at the very least that you’ll have to read the directory less often, and it could possibly be done in the background. You could reduce the need to refresh the list even further by using your systems native file update notification API (inotify on linux) to subscribe to changes to the directory.

This doesn’t seem to be possible for you, but I once solved a similiar problem by “hashing” the files into subdirectories. In my case, the challenge was to store a couple of millions images with numeric ids. I constructed the directory structure as follows:

images/[id - (id % 1000000)]/[id - (id % 1000)]/[id].jpg

This has worked well for us, and it’s the solution that I would recommend. You could do something similiar to alpha-numeric filenames by simply taking the first two letters of the filename, and then the next two letters. I’ve done this as well once, and it did the job as well.

More Related Contents:

  1. How do you determine the ideal buffer size when using FileInputStream?
  2. How can I read a large text file line by line using Java?
  3. Delete directories recursively in Java
  4. Inserting text into an existing file via Java
  5. Fastest way to sum integers in text file
  6. Write I/O file to shared network drive using credentials
  7. Fastest way of reading relatively huge byte-files in Java
  8. How do I check if a file exists in Java?
  9. Read/Write String from/to a File in Android
  10. Does use of final keyword in Java improve the performance?
  11. Why is creating a Thread said to be expensive?
  12. java.io.FileNotFoundException: the system cannot find the file specified
  13. Check if file is already open
  14. .toArray(new MyClass[0]) or .toArray(new MyClass[myList.size()])?
  15. getResourceAsStream returns null
  16. Swing animation running extremely slow
  17. How to Write text file Java
  18. Faster alternatives to Java’s reflection [closed]
  19. Is there a way in Java to determine if a path is valid without attempting to create a file?
  20. Performance difference between Java 8 lambdas and anonymous inner classes
  21. File Write – PrintStream append
  22. Modify the content of a file using Java
  23. Java BufferedReader back to the top of a text file?
  24. Strange branching performance
  25. How to discover a File’s creation time with Java?
  26. Is it possible to force an existing Java application to use no more than x cores?
  27. Java very large heap sizes [closed]
  28. Hashmap vs Array performance
  29. Why do finalizers have a “severe performance penalty”?
  30. Will using longs instead of ints benefit in 64bit java

Leave a Comment