🧱 The Bulkhead Pattern

The Bulkhead pattern is a design pattern used in software architecture to improve the resilience and fault tolerance of a system. It works by partitioning components or resources into isolated pools or "bulkheads" so that a failure or overload in one area does not cause a cascade of failures throughout the entire system. The name comes from the watertight compartments on a ship's hull, which contain flooding to a single section to prevent the ship from sinking.

⚙️ In a software context, bulkheads can be implemented in various ways, such as:

• 🧵 Thread pools: Isolating calls to different downstream services using separate, limited thread pools.
• 🗃️ Database connection pools: Assigning dedicated connection pools for different services that share a database.
• 🧩 Processes or containers: Deploying different microservices in separate containers or processes to isolate their resources.

🧱 What Is the Bulkhead Pattern?

• Purpose: Isolate parts of the system to prevent a single failure from affecting others.
• Analogy: Like watertight compartments in a ship.
• Use Case: Protect external API calls, database access, or any resource-intensive operation.

💡 Example: Bulkhead with Polly in .NET Core

A common way to implement the Bulkhead pattern in .NET Core is by using the Polly library. This example shows how to use Polly's BulkheadPolicy to isolate API calls to different downstream services.

Scenario: An e-commerce API needs to fetch product details and user reviews from separate, potentially unreliable microservices. The product details are critical for the user interface, while the reviews are secondary. We use a bulkhead to ensure that a slow or failing review service does not block all available threads and cause the entire API to become unresponsive.

1️⃣ Install NuGet packages

Install the necessary Polly packages for your project.

dotnet add package Polly
dotnet add package Microsoft.Extensions.Http.Polly

    

2️⃣ Configure bulkheads in Program.cs

Use IHttpClientFactory to apply different Bulkhead policies for different downstream services.

using Polly;

var builder = WebApplication.CreateBuilder(args);

// Bulkhead for the critical Product Service
builder.Services.AddHttpClient("productClient", client =>
{
    client.BaseAddress = new Uri("https://product-service.example.com/");
}).AddPolicyHandler(Policy.BulkheadAsync<HttpResponseMessage>(
    maxParallelization: 10, // Max 10 concurrent requests
    maxQueuingActions: 50, // Queue up to 50 requests
    onBulkheadRejectedAsync: context =>
    {
        // Log the rejection
        Console.WriteLine("Request to Product Service was rejected by bulkhead.");
        return Task.CompletedTask;
    }));

// Bulkhead for the non-critical Review Service
builder.Services.AddHttpClient("reviewClient", client =>
{
    client.BaseAddress = new Uri("https://review-service.example.com/");
}).AddPolicyHandler(Policy.BulkheadAsync<HttpResponseMessage>(
    maxParallelization: 3, // Only 3 concurrent requests
    maxQueuingActions: 10, // Queue up to 10 requests
    onBulkheadRejectedAsync: context =>
    {
        // Log the rejection and return a fallback
        Console.WriteLine("Request to Review Service was rejected by bulkhead.");
        return Task.CompletedTask;
    }));

var app = builder.Build();

// Other app configuration...

app.Run();

    

3️⃣ Use the clients in a controller

Inject IHttpClientFactory into your controller or service and create clients based on the named bulkheads.

public class ProductDetailsController : ControllerBase
{
    private readonly IHttpClientFactory _clientFactory;

    public ProductDetailsController(IHttpClientFactory clientFactory)
    {
        _clientFactory = clientFactory;
    }

    [HttpGet("product/{id}")]
    public async Task<IActionResult> GetProductDetails(int id)
    {
        // Get the client for the critical Product Service
        var productClient = _clientFactory.CreateClient("productClient");
        var reviewClient = _clientFactory.CreateClient("reviewClient");

        var productTask = productClient.GetAsync($"products/{id}");
        var reviewsTask = reviewClient.GetAsync($"reviews/{id}");

        // Wait for product details, but don't hold up for reviews
        var productResponse = await productTask;

        // The reviews task might be rejected, so handle the exception
        try
        {
            var reviewResponse = await reviewsTask;
            // Process reviews...
        }
        catch (Polly.Bulkhead.BulkheadRejectedException)
        {
            // Handle the case where the reviews request was rejected
            // Return product details without reviews
        }

        return Ok(await productResponse.Content.ReadAsStringAsync());
    }
}

    

✅ Advantages

• 🧩 Prevents cascading failures: Isolating failures within one component prevents a misbehaving dependency from impacting the entire application.
• ⚖️ Improves stability: Ensures that the system remains stable and responsive for all other operations, even when one dependency is failing or slow.
• 🎯 Fair resource allocation: Bulkheads prevent a single, poorly behaving component from monopolizing all shared resources, like threads, connections, or memory.
• 🚦 Prioritization: Allows for prioritizing access to resources. You can configure higher concurrency limits for critical services and lower limits for less important ones.

⚠️ Disadvantages

• 🧮 Increased complexity: Designing and managing separate resource pools adds complexity to the system architecture and its configuration.
• 💤 Resource underutilization: By statically partitioning resources, you might have idle resources in one bulkhead while another is under heavy load.
• 📊 Requires careful tuning: Determining the right number of concurrent calls and queue size for each bulkhead requires careful monitoring and tuning.

📅 When to use

• 🌐 Interacting with unreliable dependencies: When calling external APIs, databases, or microservices prone to latency or failure.
• 🧠 Preventing resource exhaustion: In multi-threaded applications, to protect the application's thread pool from being consumed by a failing service.
• 🔀 Prioritizing workloads: When you have critical vs. non-critical traffic that should be isolated.

🚫 When not to use

• 🏗️ For simple, monolith applications: The added complexity is not justified for apps without external dependencies.
• ❌ For permanent failures: Bulkheads protect against resource exhaustion but don't solve the root issue.
• 💪 When resources are abundant: If performance impact is minimal, simpler patterns like Retry might suffice.

🏆 Best practices

• 🔗 Combine with other resilience patterns like Circuit Breaker, Retries, and Timeouts.
• 📈 Tune, monitor, and adjust using Prometheus, Grafana, etc.
• 📝 Log bulkhead rejections for insights and fine-tuning.
• ⚙️ Use IHttpClientFactory to apply policies consistently.

🧠 Precautions

• ⚠️ Watch for dependency conflicts: Ensure proper isolation.
• 🧵 Understand semaphore vs. thread pool: Choose based on performance needs.
• 🚧 Beware of shared resources: Bulkheads fail if the resource is shared (e.g., same DB pool).

🔍 Limitations

• 🚫 Doesn't prevent the original failure: It only contains damage.
• 📉 Static resource allocation: Requires manual tuning in Polly.
• 🕒 No guarantee of availability: Prevents cascades but not recovery.

💡 Tips

• ⚙️ Start with conservative limits and adjust as needed.
• 🪄 Use fallback strategies like cached responses or defaults.
• 🎯 Prioritize critical calls: Allocate more resources to them.

Comments and Discussion