Parallel.ForEach memory usage keeps growing

by

The Parallel.ForEach method is intended for parallelizing CPU-bound workloads. Downloading a file is an I/O bound workload, and so the Parallel.ForEach is not ideal for this case because it needlessly blocks ThreadPool threads. The correct way to do it is asynchronously, with async/await. The recommended class for making asynchronous web requests is the HttpClient, and for controlling the level of concurrency an excellent option is the TPL Dataflow library. For this case it is enough to use the simplest component of this library, the ActionBlock class:

async Task DownloadListAsync(List<string> list)
{
    using (var httpClient = new HttpClient())
    {
        var rest = ExcludeDownloaded(list);
        var block = new ActionBlock<string>(async link =>
        {
            await DownloadFileAsync(httpClient, link);
        }, new ExecutionDataflowBlockOptions()
        {
            MaxDegreeOfParallelism = 10
        });
        foreach (var link in rest)
        {
            await block.SendAsync(link);
        }
        block.Complete();
        await block.Completion;
    }
}

async Task DownloadFileAsync(HttpClient httpClient, string link)
{
    var fileName = Guid.NewGuid().ToString(); // code to generate unique fileName;
    var filePath = Path.Combine(SavePath, fileName);
    if (File.Exists(filePath)) return;
    var response = await httpClient.GetAsync(link);
    response.EnsureSuccessStatusCode();
    using (var contentStream = await response.Content.ReadAsStreamAsync())
    using (var fileStream = new FileStream(filePath, FileMode.Create,
        FileAccess.Write, FileShare.None, 32768, FileOptions.Asynchronous))
    {
        await contentStream.CopyToAsync(fileStream);
    }
}

The code for downloading a file with HttpClient is not as simple as the WebClient.DownloadFile(), but it’s what you have to do in order to keep the whole process asynchronous (both reading from the web and writing to the disk).

Caveat: Asynchronous filesystem operations are currently not implemented efficiently in .NET. For maximum efficiency it may be preferable to avoid using the FileOptions.Asynchronous option in the FileStream constructor.

.NET 6 update: The preferable way for parallelizing asynchronous work is now the Parallel.ForEachAsync API. A usage example can be found here.

More Related Contents:

  1. How can I convert this foreach code to Parallel.ForEach?
  2. Multiple Parallel.ForEach loops in .Net
  3. Is this use of Parallel.ForEach() thread safe?
  4. Break parallel.foreach?
  5. .NET Core equivalent to Thread.Abort
  6. Why is Thread.Sleep so harmful
  7. Parallel.ForEach Slower than ForEach
  8. C# Thread Termination and Thread.Abort()
  9. Get the serial number of USB storage devices in .Net Core 2.1
  10. C#: Waiting for all threads to complete
  11. System.Text.Json: How do I specify a custom name for an enum value?
  12. What are the benefits to marking a field as `readonly` in C#?
  13. Set timeout to an operation
  14. Filter all queries (trying to achieve soft delete)
  15. lock keyword in C#
  16. Wait for a while without blocking main thread
  17. Using Razor outside of MVC in .NET Core
  18. Workaround for the WaitHandle.WaitAll 64 handle limit?
  19. How to validate signature of JWT from jwks without x5c
  20. Getting Cross-thread operation not valid [duplicate]
  21. Waiting for all threads to complete, with a timeout
  22. Inject service into Action Filter
  23. How to set json serializer settings in asp.net core 3?
  24. Dispatcher Invoke(…) vs BeginInvoke(…) confusion
  25. Include ONLY .xml file of SPECIFIC nuget package in a different folder
  26. Using a Scoped service in a Singleton in an Asp.Net Core app
  27. Cannot find command ‘dotnet ef’
  28. ASP.NET Core modify/substitute request body
  29. How to work threading with ConcurrentQueue
  30. Net Core: Execute All Dependency Injection in Xunit Test for AppService, Repository, etc

Leave a Comment