Async vs. Parallel Processing in C#

What is the difference between Parallel.ForEach() and Task.WhenAll()? When would you use each?

— Parallel.ForEach() vs Task.WhenAll() are both ways to run work concurrently in .NET, but they serve different purposes and have different trade-offs.

🔄 Parallel.ForEach()

• What it does: Executes a loop in parallel using multiple threads from the ThreadPool.
• Characteristics:
  • Designed for CPU-bound work (e.g., calculations, transformations).
  • Blocks the calling thread until all iterations finish.
  • Automatically manages partitioning and scheduling across available cores.
• When to use:
  • Heavy computational tasks that can be split into independent chunks.
  • Example: Image processing, mathematical simulations, data transformations.

Parallel.ForEach(items, item =>
{
    ProcessItem(item); // CPU-bound work
});

⚡ Task.WhenAll()

• What it does: Awaits completion of multiple Tasks (usually async operations).
• Characteristics:
  • Designed for I/O-bound work (e.g., database calls, HTTP requests).
  • Non-blocking — returns a Task you can await.
  • Gives you more control: you create tasks explicitly, then wait for them all.
• When to use:
  • Asynchronous operations that can run in parallel without blocking threads.
  • Example: Calling multiple APIs, querying multiple databases, reading files asynchronously.

var tasks = items.Select(item => ProcessItemAsync(item));
await Task.WhenAll(tasks); // I/O-bound work

⚖️ Key Differences

Feature	Parallel.ForEach()	Task.WhenAll()
Best for	CPU-bound work	I/O-bound work
Blocking	Yes (synchronous)	No (async/await)
Control	Limited	Full control over tasks
Thread usage	Uses ThreadPool	Uses async tasks (minimal threads)
Return values	Not directly	Collect results easily

🧩 Practical Guidance

• Use Parallel.ForEach() when you want to crunch numbers or process data in memory across multiple cores.
• Use Task.WhenAll() when you’re waiting on multiple async operations (like API calls or DB queries) and want them to run concurrently without blocking threads.

👉 A simple rule of thumb:

• CPU-bound → Parallel.ForEach()
• I/O-bound → Task.WhenAll()

Here’s a combined example showing how you might use both Task.WhenAll() and Parallel.ForEach() together in a real-world scenario:

🌐 Scenario

• You need to fetch data from multiple APIs (I/O-bound → Task.WhenAll()).
• Once the data arrives, you need to process it in parallel (CPU-bound → Parallel.ForEach()).

Example in C#

using System;
using System.Collections.Generic;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;

class Program
{
    static async Task Main()
    {
        var urls = new List<string>
        {
            "https://api.example.com/users",
            "https://api.example.com/orders",
            "https://api.example.com/payments"
        };

        using var httpClient = new HttpClient();

        // Step 1: Fetch data concurrently (I/O-bound)
        var tasks = urls.Select(url => httpClient.GetStringAsync(url));
        var responses = await Task.WhenAll(tasks);

        // Step 2: Process data in parallel (CPU-bound)
        Parallel.ForEach(responses, response =>
        {
            var processed = ProcessData(response);
            Console.WriteLine($"Processed result length: {processed.Length}");
        });
    }

    static string ProcessData(string data)
    {
        // Simulate CPU-heavy work (e.g., parsing, transformation)
        return new string(data.Reverse().ToArray());
    }
}

⚖️ Why This Works

• Task.WhenAll() → Efficiently fetches multiple API responses without blocking threads.
• Parallel.ForEach() → Maximizes CPU usage when crunching the results.

🧩 Rule of Thumb

• Use Task.WhenAll() for async I/O (network, DB, file).
• Use Parallel.ForEach() for CPU-bound processing once you have the data.

This pattern is common in data pipelines: pull data from multiple sources asynchronously, then process it in parallel for analytics, transformations, or reporting.