How to write super-fast file-streaming code in C#?

by

I don’t believe there’s anything within .NET to allow copying a section of a file without buffering it in memory. However, it strikes me that this is inefficient anyway, as it needs to open the input file and seek many times. If you’re just splitting up the file, why not open the input file once, and then just write something like:

public static void CopySection(Stream input, string targetFile, int length)
{
    byte[] buffer = new byte[8192];

    using (Stream output = File.OpenWrite(targetFile))
    {
        int bytesRead = 1;
        // This will finish silently if we couldn't read "length" bytes.
        // An alternative would be to throw an exception
        while (length > 0 && bytesRead > 0)
        {
            bytesRead = input.Read(buffer, 0, Math.Min(length, buffer.Length));
            output.Write(buffer, 0, bytesRead);
            length -= bytesRead;
        }
    }
}

This has a minor inefficiency in creating a buffer on each invocation – you might want to create the buffer once and pass that into the method as well:

public static void CopySection(Stream input, string targetFile,
                               int length, byte[] buffer)
{
    using (Stream output = File.OpenWrite(targetFile))
    {
        int bytesRead = 1;
        // This will finish silently if we couldn't read "length" bytes.
        // An alternative would be to throw an exception
        while (length > 0 && bytesRead > 0)
        {
            bytesRead = input.Read(buffer, 0, Math.Min(length, buffer.Length));
            output.Write(buffer, 0, bytesRead);
            length -= bytesRead;
        }
    }
}

Note that this also closes the output stream (due to the using statement) which your original code didn’t.

The important point is that this will use the operating system file buffering more efficiently, because you reuse the same input stream, instead of reopening the file at the beginning and then seeking.

I think it’ll be significantly faster, but obviously you’ll need to try it to see…

This assumes contiguous chunks, of course. If you need to skip bits of the file, you can do that from outside the method. Also, if you’re writing very small files, you may want to optimise for that situation too – the easiest way to do that would probably be to introduce a BufferedStream wrapping the input stream.

More Related Contents:

  1. How to create a new object instance from a Type
  2. Is a LINQ statement faster than a ‘foreach’ loop?
  3. Comparing two byte arrays in .NET
  4. Do try/catch blocks hurt performance when exceptions are not thrown?
  5. Handling warning for possible multiple enumeration of IEnumerable
  6. Excel Interop – Efficiency and performance
  7. C# ‘is’ operator performance
  8. Is there a performance impact when calling ToList()?
  9. Performance difference for control structures ‘for’ and ‘foreach’ in C#
  10. Performance Tests of Serializations used by WCF Bindings
  11. Performance of static methods vs instance methods
  12. HttpWebRequest is extremely slow!
  13. When is optimization premature? [closed]
  14. Advantages to Using Private Static Methods
  15. Efficient Cartesian Product algorithm
  16. Static vs. non-static method
  17. What’s the difference between encapsulating a private member as a property and defining a property without a private member?
  18. Performance of “direct” virtual call vs. interface call in C#
  19. C# quickest way to shift array
  20. System.Net.WebClient unreasonably slow
  21. How much faster is C++ than C#?
  22. Console.WriteLine slow
  23. How do I measure how long a function is running?
  24. DateTime.DayOfWeek micro optimization
  25. Which is faster: x
  26. How to read last “n” lines of log file [duplicate]
  27. A better way to replace many strings – obfuscation in C#
  28. Get all files and directories in specific path fast
  29. Performance of compiled-to-delegate Expression
  30. Fastest implementation of log2(int) and log2(float)

Leave a Comment