A practical pattern for preventing cascading failures, coordinating state across replicas, and serving controlled fallbacks.

By Reshma Thakkallapelly · 7–9 minute read

A slow or unavailable dependency should not be allowed to consume every thread, connection, and retry budget in a microservices platform.

I built a distributed circuit-breaker layer that uses Redis for fast shared state and MongoDB for durable configuration, approved fallback snapshots, and state-transition history.

Fail fast when a dependency is unhealthy, recover cautiously, and degrade without hiding the failure.

The Failure Pattern I Needed to Stop

In a replicated microservices environment, one unhealthy downstream API can trigger a cascading failure.

Every service instance continues sending traffic. Application retries multiply the load. Connection pools fill up. Request queues grow, and response times increase across otherwise healthy services.

An in-process circuit breaker can protect one service instance. For this use case, I also needed:

• A coordinated view of dependency health across replicas
• Centrally managed failure thresholds and cooldown policies
• Controlled recovery probes
• Safe and explicit fallback behavior
• A durable record of state transitions
• Operational visibility into why calls were allowed, rejected, or degraded

I therefore separated the design into three planes.

Fast Data Plane

The microservice and Redis decide whether a remote call is permitted. The decision must remain fast and must not depend on a MongoDB read for every request.

Durable Control Plane

MongoDB stores versioned configuration, transition history, approved fallback snapshots, and governance information.

Observability Plane

Metrics, logs, and traces explain why calls were permitted, rejected, retried, or served through a fallback.

Reference Architecture

The normal request path stays intentionally small. MongoDB is not a mandatory synchronous dependency for every request. Configuration is warmed into Redis, and transition records are persisted asynchronously.

Circuit-Breaker Interceptor

The interceptor executes before and after the remote call.

Redis

Redis stores the current breaker state, failure counters, cooldown TTL, half-open probe lock, and hot fallback data.

MongoDB

MongoDB stores versioned policy, state-transition history, operational annotations, and durable fallback snapshots.

Remote Client

The remote client calls the downstream dependency.

Fallback Handler

The fallback handler returns an approved cached response, a static safe default, a deferred-processing response, or a controlled failure.

Telemetry

Telemetry records breaker state, rejections, fallback use, latency, errors, and recovery behavior.

How a Request Moves Through the Breaker

1. Check Shared State

The interceptor reads the dependency-specific state from Redis.

At high throughput, a very short local cache may be used for the common CLOSED state, provided that the design defines how quickly an OPEN transition must propagate.

2. Decide

The current state determines whether the dependency call is permitted:

• CLOSED permits the call.
• OPEN rejects the call immediately.
• HALF_OPEN permits only a small number of controlled probe calls.

A Redis lock or permit counter coordinates half-open probes across replicas.

3. Call with a Deadline

The remote client applies a short timeout.

Retries are bounded, use only the caller’s available time budget, and are included in the circuit breaker’s failure accounting.

4. Update State Atomically

A successful call resets or gradually decays the failure state.

A qualifying dependency failure increments the failure counter. When the configured threshold is crossed, the circuit changes to OPEN and starts its cooldown TTL.

The state evaluation and transition should occur atomically.

5. Degrade Safely

When the dependency call is rejected or fails, the fallback handler applies an explicitly approved strategy.

Depending on the use case, the response may be:

• A cached value
• A static safe default
• A deferred-processing result such as HTTP 202 Accepted
• A fast and precise controlled error

6. Persist and Observe

State transitions are emitted asynchronously for durable MongoDB history, operational dashboards, alerting, and incident analysis.

The Three Circuit-Breaker States

Why the Half-Open Lock Matters

Without a distributed permit, every service replica may probe the recovering dependency at the same moment after the cooldown expires.

That recovery surge can overload a service that has only just started becoming healthy.

A Redis SET NX key with a short TTL, or an atomic permit counter, limits recovery traffic and prevents a probe storm.

Redis Key Model

I scope each circuit breaker by the calling service and downstream dependency so that unrelated integrations do not share failure state.

cb:{caller}:{dependency}:state
cb:{caller}:{dependency}:failures
cb:{caller}:{dependency}:opened_at
cb:{caller}:{dependency}:half_open_permit
fallback:{dependency}:{business_key}

The state update should be atomic.

A Redis Lua script can read the current state, increment the qualifying failure count, compare it with the threshold, change the state to OPEN, and assign the cooldown TTL as one operation.

This prevents race conditions when several service instances experience failures at the same time.

MongoDB as the Durable Control Plane

MongoDB stores information that must survive Redis eviction, restart, policy changes, or environment reconfiguration.

The durable control plane may include:

• Versioned circuit-breaker policies by caller and dependency
• State-transition history containing the timestamp, reason, previous state, new state, and correlation ID
• Approved fallback snapshots when the use case permits last-known-good data
• Manual override records
• Configuration expiry information
• Policy ownership and approval information
• Operational notes used during incident review

An illustrative configuration document might look like this:

{
  "_id": "claims-service:member-service",
  "failureThreshold": 5,
  "windowSeconds": 30,
  "openSeconds": 20,
  "halfOpenPermits": 1,
  "fallbackPolicy": "CACHE_THEN_DEFER",
  "version": 7,
  "enabled": true
}

Illustrative Service Logic

The implementation is framework-neutral. The following Java-style pseudocode illustrates the responsibility boundary:

public Response execute(Request request) {
    Decision decision = breaker.beforeCall("member-service");

    if (!decision.allowed()) {
        return fallback.resolve(request, decision.state());
    }

    try {
        Response response = client.call(request, SHORT_TIMEOUT);
        breaker.onSuccess("member-service");
        return response;

    } catch (TimeoutException | ServiceUnavailableException ex) {
        breaker.onFailure("member-service", ex);

        return fallback.resolve(
            request,
            breaker.currentState()
        );
    }
}

Only failures that indicate dependency health should count toward opening the circuit.

Business-validation errors, authentication failures caused by the caller, and expected client-side 4xx responses generally require separate treatment.

Fallbacks Must Be Explicit

A circuit breaker protects system capacity. The fallback protects the user journey.

I use one of four explicit fallback strategies.

Cached Response

Best fit: Reference data or read-heavy information for which an approved last-known value remains useful.

Guardrail: Include the source and freshness timestamp. Do not represent stale data as current.

Static Safe Default

Best fit: Non-critical feature flags, optional enrichment, or behavior that can safely be omitted.

Guardrail: Never invent a business, financial, clinical, eligibility, or authorization decision.

Deferred Processing

Best fit: Work that may continue asynchronously.

Guardrail: Return a tracking identifier, explain the current status clearly, and provide a reliable way to retrieve the final result.

Controlled Failure

Best fit: Situations in which no safe substitute exists.

Guardrail: Return a precise error quickly. Do not allow the request to hang until every timeout is exhausted.

Observability and Operational Signals

A circuit breaker is useful only when operators can explain its behavior.

My minimum telemetry set includes:

• Current state by calling service and dependency
• Number of allowed calls
• Number of rejected calls
• Number of fallback invocations
• Failure count and failure rate
• State-transition count
• Downstream latency and timeout rate
• Half-open probe outcomes
• Redis latency and command failures
• Half-open permit-lock contention
• Age of cached fallback data
• Percentage of requests served with degraded data
• Business transactions delayed, deferred, or rejected

The most useful alert is not simply:

“Circuit open.”

A more actionable alert is:

“Circuit open for a critical dependency, with a rising degraded-response rate and measurable business impact.”

Production Guardrails

Protect Redis

Use multi-zone replication or a managed Redis service, strict command timeouts, and a clearly defined behavior for situations in which Redis itself is unavailable.

Avoid a Breaker Avalanche

Scope breaker keys carefully.

One noisy tenant, endpoint, operation, or downstream capability should not open an unnecessarily broad circuit for unrelated traffic.

Coordinate Retries

Retry only failures that are genuinely transient.

Use exponential backoff with jitter, cap the number of attempts, and ensure that retries do not exceed the caller’s remaining time budget.

Keep Recovery Probes Scarce

Allow only a small number of half-open calls.

Every probe lock or permit should have an expiration so that a failed or abandoned probe cannot block recovery indefinitely.

Secure the Data

Do not place sensitive identifiers in Redis keys, logs, metrics, or traces.

Encrypt fallback payloads where required, limit access, and expire data according to the applicable retention policy.

Version Configuration

Use optimistic versioning for policy updates.

Record and audit every manual override, policy change, emergency action, and rollback.

Test Failure Modes

Test more than the successful path.

Simulate:

• Dependency timeouts
• Connection resets
• Redis latency
• Redis unavailability
• MongoDB unavailability
• Stale fallback data
• Simultaneous recovery across multiple service replicas
• Probe-lock contention
• Retry amplification
• Configuration rollback
• Expired or invalid fallback snapshots

An Important Trade-Off

A distributed circuit breaker improves coordination, but it also introduces Redis as an additional dependency.

For a lower-scale system, an in-process circuit breaker may be simpler and entirely sufficient.

For a high-throughput replicated platform, a hybrid design—using a local fast path together with Redis-backed coordination—often provides a better balance between latency, availability, and consistent recovery behavior.

What This Pattern Achieves

Final Thoughts

A circuit breaker is not merely an exception handler.

It is a state machine that determines when a microservice is permitted to spend capacity on an unhealthy dependency.

In this implementation, Redis provides fast and atomic coordination across service replicas. MongoDB provides durable configuration, fallback governance, and operational history.

Separating these responsibilities keeps the request path responsive while preserving enough evidence to tune thresholds, investigate incidents, and improve recovery policies.

A good circuit breaker does three things well: it stops damage quickly, tests recovery carefully, and makes degradation visible.