Load Shedding: Dropping Requests to Save Your System

Your system under pressure will either shed load deliberately - or collapse accidentally.

Traffic spikes. Response times climb. Clients start retrying. The retries add more load. Response times climb further. At some point, the whole thing tips over, and every user gets nothing instead of something.

Most teams I talk to reach for circuit breakers or rate limiters when this happens. Both are good tools. Neither solves this problem. The missing piece - the one that keeps your system alive when everything else is failing - is load shedding.

And the reason most teams don't have it is that they think they already do.

What load shedding actually is (and what it isn't) 🔗

Let me be precise about terms, because this is where most confusion starts.

Rate limiting is about identity and quota. You allow 100 requests per second per client. It doesn't matter how healthy your system is - if a client is over quota, they're rejected. Rate limiting protects you from individual bad actors or runaway clients.

Circuit breakers are about outbound calls. When a downstream service is failing, you stop calling it. The circuit breaker sits between you and your dependencies.

Load shedding is different. It responds to your system's health right now. When CPU is saturating, memory is tight, or your request queue is backing up - you drop requests regardless of who sent them. Not because of policy. Because you physically cannot serve them well.

The correct HTTP response for a shed request is 503 Service Unavailable with a Retry-After header. Not 429 Too Many Requests - that implies the caller is the problem. 503 says: it's us, not you, try again in a moment.

A minimal implementation in ASP.NET Core 7+ looks like this:

app.Use(async (context, next) =>
{
    var memoryInfo = GC.GetGCMemoryInfo();
    var load = (double)memoryInfo.MemoryLoadBytes / memoryInfo.HighMemoryLoadThresholdBytes;

    if (load > 0.85 || _activeRequests > _maxConcurrent)
    {
        context.Response.StatusCode = 503;
        context.Response.Headers["Retry-After"] = "5";
        return;
    }

    Interlocked.Increment(ref _activeRequests);
    try { await next(context); }
    finally { Interlocked.Decrement(ref _activeRequests); }
});

Simple. But where you put this matters more than you might think.

The cost of where you shed 🔗

Think about it: by the time your middleware runs, your system has already done a lot of work.

The TCP connection was accepted. TLS was negotiated. HTTP/2 frames were parsed. Request headers were read. The ASP.NET Core pipeline started. Middleware before yours executed. Dependency injection resolved your services. Authentication ran.

All of that happened before you decided to return a 503.

Imagine a restaurant that seats guests, hands them menus, takes their drink order, and sends it to the kitchen - and only then tells them the kitchen is closed. You've wasted everyone's time, including your own. The earlier you shed, the cheaper each rejection is.

Kestrel exposes MaxConcurrentConnections and MaxConcurrentUpgradedConnections for connection-level limiting - before any HTTP parsing happens. That's the cheapest possible rejection. Your custom middleware is next. Your controller is the most expensive place to shed, because by then you've done almost everything.

builder.WebHost.ConfigureKestrel(options =>
{
    options.Limits.MaxConcurrentConnections = 1000;
});

In practice, you often need both: Kestrel limits as a hard ceiling, middleware for health-signal-based shedding. The point is to push the decision as far left in the stack as you can.

Not all requests deserve equal treatment 🔗

Here's where most load shedding implementations leave a lot on the table.

Standard shedding is binary - you're over threshold, so you reject the next request in the queue. But think about what's actually in that queue. Health check probes from your orchestrator. Background telemetry events. An analytics batch job. And a user trying to complete a payment.

These are not the same. Dropping a telemetry flush is free. Dropping a payment request has a real cost.

Google's SRE team calls these criticality labels - not all RPCs are equal, so you shouldn't shed them equally. In .NET 7+, you can implement the same idea with endpoint metadata and a priority-aware middleware.

public class SheddingPriorityAttribute : Attribute, IEndpointMetadata
{
    public SheddingPriority Priority { get; }
    public SheddingPriorityAttribute(SheddingPriority priority) => Priority = priority;
}

public enum SheddingPriority { ShedFirst, Normal, Protected }

Then in your middleware, read the priority before deciding whether to shed:

app.Use(async (context, next) =>
{
    if (IsOverloaded())
    {
        var priority = context.GetEndpoint()
            ?.Metadata.GetMetadata<SheddingPriorityAttribute>()
            ?.Priority ?? SheddingPriority.Normal;

        if (priority != SheddingPriority.Protected)
        {
            context.Response.StatusCode = 503;
            context.Response.Headers["Retry-After"] = "5";
            return;
        }
    }

    await next(context);
});

Apply it on your endpoints:

app.MapGet("/metrics", ...).WithMetadata(new SheddingPriorityAttribute(SheddingPriority.ShedFirst));
app.MapPost("/checkout", ...).WithMetadata(new SheddingPriorityAttribute(SheddingPriority.Protected));

Uber described something similar in 2019 - they shed roughly 10% of requests during peak events with no measurable customer impact. The reason? They shed the right 10%.

Static thresholds will betray you 🔗

Most implementations I see use a static threshold: if CPU > 80%, start shedding. In my opinion, this is better than nothing - but it's a guess, and often a late one.

Here's why. Queueing theory (the M/M/1 model, if you want to look it up) tells us that latency doesn't degrade linearly as utilisation climbs. It degrades super-linearly. At 70% utilisation things feel fine. At 85% they start to slip. By 95% you're already in trouble - and that's when the threshold fires.

The smarter signal isn't the absolute CPU value. It's the gradient - is latency getting worse? By how much, and how fast?

Netflix's Concurrency Limits library, which powers parts of their resilience infrastructure, uses TCP Vegas for exactly this. TCP Vegas was designed for network congestion control - instead of detecting packet loss (the cliff), it detects the onset of congestion by tracking latency increase. Load shedding can work the same way.

There's no standard .NET library that does adaptive shedding like this. It's a genuine gap in the ecosystem. But you can get close with System.Diagnostics.Metrics in .NET 8 - track a rolling P99 latency histogram and engage shedding when the gradient turns positive:

var histogram = meter.CreateHistogram<double>("request.duration.ms");

// Record on every request: histogram.Record(elapsedMs);
// In your shedding middleware: compare rolling P99 against baseline
// When P99 rises >20% over baseline, start shedding

Start shedding when things are getting worse, not when they're already broken. That's the difference between a controlled degradation and a collapse.

What this means in practice 🔗

Load shedding isn't one decision - it's a stack of decisions. Here's how I'd approach it:

• Shed at the lowest layer you can - Kestrel limits before middleware, middleware before controllers
• Assign criticality to endpoint groups - protect payments, shed telemetry and batch jobs first
• Use health signals, not static thresholds - track latency gradient, not just CPU percentage
• Return 503 + Retry-After, never 429 for load shedding - the semantics matter
• Test it under load - simulate overload in staging and verify that P99 for accepted requests actually improves; if it doesn't, your threshold is wrong
• Layer it with your other resilience tools - load shedding doesn't replace rate limiting, circuit breakers, or backpressure; it completes them

Remember: properly implemented load shedding improves the experience for the requests you do accept. By refusing to queue work you can't process quickly, you keep latency tight for everything you take on. AWS and Google both document this. It's counterintuitive until you see the numbers.

Load shedding feels like giving up. It feels like admitting your system can't handle the load, that you're turning users away.

In reality, it's the opposite. It's your system making a deliberate, informed choice about who it can serve well right now - instead of trying to serve everyone badly and tipping into total failure. Systems that shed deliberately stay alive. Systems that don't shed eventually serve no one.

The question worth asking: has a retry storm ever taken down one of your services? Because load shedding is precisely the thing that would have stopped it.

If you find this useful, ping me on Twitter/X or LinkedIn - and a follow there is always appreciated if you'd like more of this. I'm always curious what resilience patterns teams are actually using in production.

load-shedding aspnet-core kestrel dotnet-7 dotnet-8 resilience