Nesting await in Parallel.ForEach [duplicate]

by

The whole idea behind Parallel.ForEach() is that you have a set of threads and each thread processes part of the collection. As you noticed, this doesn’t work with async–await, where you want to release the thread for the duration of the async call.

You could “fix” that by blocking the ForEach() threads, but that defeats the whole point of async–await.

What you could do is to use TPL Dataflow instead of Parallel.ForEach(), which supports asynchronous Tasks well.

Specifically, your code could be written using a TransformBlock that transforms each id into a Customer using the async lambda. This block can be configured to execute in parallel. You would link that block to an ActionBlock that writes each Customer to the console.
After you set up the block network, you can Post() each id to the TransformBlock.

In code:

var ids = new List<string> { "1", "2", "3", "4", "5", "6", "7", "8", "9", "10" };

var getCustomerBlock = new TransformBlock<string, Customer>(
    async i =>
    {
        ICustomerRepo repo = new CustomerRepo();
        return await repo.GetCustomer(i);
    }, new ExecutionDataflowBlockOptions
    {
        MaxDegreeOfParallelism = DataflowBlockOptions.Unbounded
    });
var writeCustomerBlock = new ActionBlock<Customer>(c => Console.WriteLine(c.ID));
getCustomerBlock.LinkTo(
    writeCustomerBlock, new DataflowLinkOptions
    {
        PropagateCompletion = true
    });

foreach (var id in ids)
    getCustomerBlock.Post(id);

getCustomerBlock.Complete();
writeCustomerBlock.Completion.Wait();

Although you probably want to limit the parallelism of the TransformBlock to some small constant. Also, you could limit the capacity of the TransformBlock and add the items to it asynchronously using SendAsync(), for example if the collection is too big.

As an added benefit when compared to your code (if it worked) is that the writing will start as soon as a single item is finished, and not wait until all of the processing is finished.

More Related Contents:

  1. Parallel foreach with asynchronous lambda
  2. Parallel.ForEach and async-await [duplicate]
  3. await vs Task.Wait – Deadlock?
  4. An async/await example that causes a deadlock
  5. Catch an exception thrown by an async void method
  6. Awaiting multiple Tasks with different results
  7. Using async/await for multiple tasks
  8. How to safely call an async method in C# without await
  9. General purpose FromEvent method
  10. I want await to throw AggregateException, not just the first Exception
  11. Parallel.ForEach vs Task.Run and Task.WhenAll
  12. Enforce an async method to be called once
  13. Is there anything like asynchronous BlockingCollection?
  14. Cancelling a pending task synchronously on the UI thread
  15. What is the use for Task.FromResult in C#
  16. ConfigureAwait pushes the continuation to a pool thread
  17. How can I prevent synchronous continuations on a Task?
  18. Running async methods in parallel
  19. Different forms of the WCF service contract interface
  20. Accessing UI controls in Task.Run with async/await on WinForms
  21. Task.Factory.StartNew with async lambda and Task.WaitAll
  22. What is the correct way to cancel an async operation that doesn’t accept a CancellationToken?
  23. Faulted vs Canceled task status after CancellationToken.ThrowIfCancellationRequested
  24. Synchronous implementation of interface that returns Task
  25. How do I get a return value from Task.WaitAll() in a console app?
  26. Task.WhenAny – What happens with remaining running tasks?
  27. Transform IEnumerable asynchronously by awaiting each task
  28. How to implement Task.WhenAny() with a predicate
  29. Create a completed Task
  30. Calling TaskCompletionSource.SetResult in a non blocking manner

Leave a Comment