.net Core Parallel.ForEach issues

by

Why Parallel.ForEach is not good for this task is explained in comments: it’s designed for CPU-bound (CPU-intensive) tasks. If you use it for IO-bound operations (like making web requests) – you will waste thread pool thread blocked while waiting for response, for nothing good. It’s possible to use it still, but it’s not best for this scenario.

What you need is to use asynchronous web request methods (like HttpWebRequest.GetResponseAsync), but here comes another problem – you don’t want to execute all your web requests at once (like another answer suggests). There may be thousands urls (ids) in your list. So you can use thread synchronization constructs designed for that, for example Semaphore. Semaphore is like queue – it allows X threads to pass, and the rest should wait until one of busy threads will finish it’s work (a bit simplified description). Here is an example:

static async Task ProcessUrls(string[] urls) {
    var tasks = new List<Task>();
    // semaphore, allow to run 10 tasks in parallel
    using (var semaphore = new SemaphoreSlim(10)) {
        foreach (var url in urls) {
            // await here until there is a room for this task
            await semaphore.WaitAsync();
            tasks.Add(MakeRequest(semaphore, url));
        }
        // await for the rest of tasks to complete
        await Task.WhenAll(tasks);
    }
}

private static async Task MakeRequest(SemaphoreSlim semaphore, string url) {
    try {
        var request = (HttpWebRequest) WebRequest.Create(url);

        using (var response = await request.GetResponseAsync().ConfigureAwait(false)) {
            // do something with response    
        }
    }
    catch (Exception ex) {
        // do something
    }
    finally {
        // don't forget to release
        semaphore.Release();
    }
}

More Related Contents:

  1. how to use c# For(;;) in parallel [closed]
  2. What is the difference between asynchronous programming and multithreading?
  3. How to abort a Task like aborting a Thread (Thread.Abort method)?
  4. Why should I prefer single ‘await Task.WhenAll’ over multiple awaits?
  5. No ConcurrentList in .Net 4.0?
  6. Does Parallel.ForEach limit the number of active threads?
  7. How to put a task to sleep (or delay) in C# 4.0?
  8. How to run unit tests (MSTest) in parallel?
  9. How expensive is the lock statement?
  10. Do the new C# 5.0 ‘async’ and ‘await’ keywords use multiple cores?
  11. Parallel Sort Algorithm
  12. Limit the number of parallel threads in C#
  13. Accelerating bitmap grayscale conversion, is OpenMP an option in C#?
  14. Parallel.For(): Update variable outside of loop
  15. Process files concurrently and asynchronously
  16. Why is Parallel.ForEach much faster then AsParallel().ForAll() even though MSDN suggests otherwise?
  17. Design help for parallel processing Azure blob and bulk copy to SQL database
  18. Is this use of Parallel.ForEach() thread safe?
  19. Simplest way to run three methods in parallel in C#
  20. Parallel.ForEach with adding to list
  21. Should thread-safe class have a memory barrier at the end of its constructor?
  22. Parallel doesnt work with Entity Framework
  23. Break parallel.foreach?
  24. increment a count value outside parallel.foreach scope
  25. How to properly parallelise job heavily relying on I/O
  26. Are C# structs thread safe?
  27. Partition: How to add a wait after every partition
  28. Wait for QueueUserWorkItem to Complete
  29. Run async method 8 times in parallel
  30. How do static properties work in an asp.net environment?

Leave a Comment