🛑 How to implement throttling in .NET Core

You can implement throttling using the queuing mechanisms available in the Microsoft.AspNetCore.RateLimiting middleware. When a request exceeds a policy's limit, it is placed in a queue to wait for a permit to become available.

🧪 Example: Fixed window throttling

This example shows how to configure a fixed window policy that queues requests instead of immediately rejecting them.

1️⃣ Register the rate limiting service with a queue:

In your Program.cs, define a policy with a QueueLimit greater than zero. When the PermitLimit is exceeded, incoming requests are placed in the queue.

using System.Threading.RateLimiting;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddRateLimiter(_ =>
{
    _.RejectionStatusCode = StatusCodes.Status503ServiceUnavailable;
    _.AddFixedWindowLimiter(policyName: "fixed-queue", options =>
    {
        options.PermitLimit = 4; // Allow 4 requests per window
        options.Window = TimeSpan.FromSeconds(12); // Window size is 12 seconds
        options.QueueProcessingOrder = QueueProcessingOrder.OldestFirst;
        options.QueueLimit = 2; // Queue up to 2 excess requests
    });
});

var app = builder.Build();

app.UseRateLimiter(); // Enable the middleware
// ...
    

2️⃣ Apply the policy to an endpoint:

Apply the policy to an endpoint using the RequireRateLimiting() extension method (Minimal APIs) or the [EnableRateLimiting("policy-name")] attribute (controllers).

app.MapGet("/", () => Results.Ok("Hello from a throttled endpoint"))
   .RequireRateLimiting("fixed-queue");
    

With this configuration, if 6 requests arrive in a 12-second window, the first 4 are handled immediately, the next 2 are queued, and any further requests are rejected with a 503 Service Unavailable status code.

🔄 Throttle algorithms with queuing

• 🧮 Fixed window: Queues requests that arrive after the window's permit limit has been reached, processing them in a new window.
• 📆 Sliding window: Similar to the fixed window, but allows queuing based on a sliding interval.
• 🪣 Token bucket: Queues requests when the token bucket is empty.
• 🔗 Concurrency: Queues requests when the number of concurrent operations exceeds the permit limit.

✅ Advantages

• 😊 Smoother user experience: Throttling queues and processes requests rather than failing them, avoiding abrupt errors for users during temporary spikes.
• 🛡️ Prevents cascading failures: It protects dependencies like databases from being overloaded by controlling the rate of requests, preventing service disruption.
• 📊 Manages system load: It helps smooth out unexpected traffic bursts, ensuring that your backend can handle the load predictably.
• 🎯 Prioritizes requests: The QueueProcessingOrder setting allows you to prioritize how queued requests are handled, with options for OldestFirst and NewestFirst.

⚠️ Disadvantages

• 🐢 Increased latency: Queuing requests introduces artificial latency, which may not be suitable for time-sensitive or real-time applications.
• 💾 Resource consumption: Maintaining a queue of pending requests consumes server memory and resources.
• 🔁 Complex client behavior: Clients need to handle the waiting period gracefully rather than immediately retrying, which could worsen the load.
• 🚨 Misconfiguration risks: An improperly sized queue or aggressive QueueLimit can still lead to service degradation or rejections.

💡 Best practices and tips

• 🧭 Queue for non-critical requests: Use throttling with queuing for less critical or long-running operations where latency is acceptable. For critical, latency-sensitive endpoints, aggressive rate limiting may be more appropriate.
• 📐 Right-size your queue: Set a reasonable QueueLimit to prevent resource exhaustion while still accommodating a manageable number of excess requests.
• 📢 Provide client feedback: While queuing avoids a 429 rejection, you can use headers to inform the client that their request has been throttled, so they don't immediately initiate another request.
• 📈 Monitor queue health: Monitor the size of your queues and queue processing times to ensure they are not growing uncontrollably, which can indicate a performance bottleneck.
• 🧪 Test under load: Perform stress testing to find the optimal balance between PermitLimit, Window, and QueueLimit that protects your resources while providing acceptable performance.

🚨 Precautions

• 🌐 Use distributed cache for scale: The default in-memory queuing is not suitable for applications running on multiple instances. In a distributed environment, use a shared, distributed cache like Redis to store and manage the rate limiting counters and queues.
• 🛠️ Throttling isn't a silver bullet: Throttling is not a substitute for a properly scaled and resilient application architecture. It should be used as a front-line defense, not as a fix for underlying performance issues.
• ⚖️ Beware of queue processing order: Be mindful of the QueueProcessingOrder. Using NewestFirst for request processing could starve older requests and be unfair to users. For most cases, OldestFirst is the better choice.

⚖️ Rate limiting vs. Throttling

While the terms "rate limiting" and "throttling" are often used interchangeably, they have subtle differences.