Reading large text files with streams in C#

by

You can improve read speed by using a BufferedStream, like this:

using (FileStream fs = File.Open(path, FileMode.Open, FileAccess.Read, FileShare.ReadWrite))
using (BufferedStream bs = new BufferedStream(fs))
using (StreamReader sr = new StreamReader(bs))
{
    string line;
    while ((line = sr.ReadLine()) != null)
    {

    }
}

March 2013 UPDATE

I recently wrote code for reading and processing (searching for text in) 1 GB-ish text files (much larger than the files involved here) and achieved a significant performance gain by using a producer/consumer pattern. The producer task read in lines of text using the BufferedStream and handed them off to a separate consumer task that did the searching.

I used this as an opportunity to learn TPL Dataflow, which is very well suited for quickly coding this pattern.

Why BufferedStream is faster

A buffer is a block of bytes in memory used to cache data, thereby reducing the number of calls to the operating system. Buffers improve read and write performance. A buffer can be used for either reading or writing, but never both simultaneously. The Read and Write methods of BufferedStream automatically maintain the buffer.

December 2014 UPDATE: Your Mileage May Vary

Based on the comments, FileStream should be using a BufferedStream internally. At the time this answer was first provided, I measured a significant performance boost by adding a BufferedStream. At the time I was targeting .NET 3.x on a 32-bit platform. Today, targeting .NET 4.5 on a 64-bit platform, I do not see any improvement.

Related

I came across a case where streaming a large, generated CSV file to the Response stream from an ASP.Net MVC action was very slow. Adding a BufferedStream improved performance by 100x in this instance. For more see Unbuffered Output Very Slow

More Related Contents:

  1. An elegant way to consume (all bytes of a) BinaryReader?
  2. How do I save a stream to a file in C#?
  3. Does disposing streamreader close the stream?
  4. How to read a large (1 GB) txt file in .NET?
  5. How do I convert struct System.Byte byte[] to a System.IO.Stream object in C#?
  6. Should I call Close() or Dispose() for stream objects?
  7. “The given path’s format is not supported.”
  8. How to find and replace text in a file
  9. What is the fastest way to create a checksum for large files in C#
  10. Stream.Seek(0, SeekOrigin.Begin) or Position = 0
  11. How do I concatenate two System.IO.Stream instances into one?
  12. HttpWebRequest & Native GZip Compression
  13. Adding a Line to the Middle of a File with .NET
  14. When or if to Dispose HttpResponseMessage when calling ReadAsStreamAsync?
  15. How to get a MemoryStream from a Stream in .NET?
  16. How can I split (copy) a Stream in .NET?
  17. Paging with LINQ for objects
  18. Read XML Attribute using XmlDocument
  19. WPF WebBrowser control – how to suppress script errors?
  20. How do I use OpenFileDialog to select a folder?
  21. C# – Making a Process.Start wait until the process has start-up
  22. FtpWebRequest FTP download with ProgressBar
  23. C# – how to determine whether a Type is a number
  24. HttpContext.Current.Session is null in Ashx file
  25. How do I clear System.Net client DNS cache?
  26. RegEx – reusing subexpressions
  27. Domain credentials for a WebClient class don’t work
  28. Entity Framework: Duplicate Records in Many-to-Many relationship
  29. How to convert DataTable to class Object?
  30. Make a form’s background transparent

Leave a Comment