File I/O with streams – best memory buffer size

by

Files are already buffered by the file system cache. You just need to pick a buffer size that doesn’t force FileStream to make the native Windows ReadFile() API call to fill the buffer too often. Don’t go below a kilobyte, more than 16 KB is a waste of memory and unfriendly to the CPU’s L1 cache (typically 16 or 32 KB of data).

4 KB is a traditional choice, even though that will exactly span a virtual memory page only ever by accident. It is difficult to profile; you’ll end up measuring how long it takes to read a cached file. Which runs at RAM speeds, 5 gigabytes/sec and up if the data is available in the cache. It will be in the cache the second time you run your test, and that won’t happen in a production environment too often. File I/O is completely dominated by the disk drive or the NIC and is glacially slow, copying the data is peanuts. 4 KB will work fine.

More Related Contents:

  1. C# FileStream : Optimal buffer size for writing large files?
  2. How to download a file from a URL in C#?
  3. Free file locked by new Bitmap(filePath)
  4. Is shifting bits faster than multiplying and dividing in Java? .NET? [closed]
  5. How to compare 2 files fast using .NET?
  6. Why always close Database connection?
  7. How to detect a USB drive has been plugged in?
  8. In C# check that filename is *possibly* valid (not that it exists) [duplicate]
  9. How to find the extension of a file in C#?
  10. GCC C++ “Hello World” program -> .exe is 500kb big when compiled on Windows. How can I reduce its size?
  11. Fake-scrolling containers with very many controls
  12. Pipe buffer size is 4k or 64k?
  13. Read last line of text file
  14. Does reactive extensions support rolling buffers?
  15. Retrieving files from directory that contains large amount of files
  16. Selecting the size of a System.Drawing.Icon?
  17. How to insert characters to a file using C#
  18. WPF/C#: Where should I be saving user preferences files?
  19. How do I get the directory from a file’s full path?
  20. Writing FizzBuzz
  21. Sending Files using HTTP POST in c# [closed]
  22. Getting files by creation date in .NET
  23. Resize text size of a label when the text gets longer than the label size?
  24. How do I detect when a directory or file changes without constant scanning
  25. A better way to replace many strings – obfuscation in C#
  26. Image file copy, is being used by another process
  27. Delete files older than 3 months old in a directory using .NET
  28. using static Regex.IsMatch vs creating an instance of Regex
  29. Creating download link to a file on a file server
  30. Find a file within all possible folders?

Leave a Comment