Circuit Breaker Pattern: Building Resilient Systems

Table of Contents

• Introduction
• The Problem Circuit Breakers Solve
• How a Circuit Breaker Works
• Closed, Open, and Half-Open States
• Example: A Struggling Recommendation Service
• Circuit Breaker vs Retry
• Fallback Strategies
• What to Monitor
• Common Mistakes
• Checklist
• FAQ
• Conclusion

Introduction

Distributed systems fail in messy ways. A payment provider slows down. A recommendation service returns errors. A database replica becomes unavailable. If every caller keeps retrying aggressively, one failure can spread across the system.

The circuit breaker pattern helps stop that spread.

It watches calls to a dependency. When failures cross a threshold, it temporarily stops sending traffic to that dependency and returns a fallback or fast failure instead.

The Problem Circuit Breakers Solve

Without a circuit breaker, callers can overload a struggling service.

Situation	Without circuit breaker
dependency is slow	threads and connections pile up
dependency returns errors	callers keep retrying
provider outage	user requests wait until timeout
partial failure	failure spreads to other services

The goal is not to hide failure. The goal is to fail fast, protect resources, and give the dependency time to recover.

How a Circuit Breaker Works

A circuit breaker tracks recent outcomes.

Example signals:

• failure rate
• timeout rate
• slow call rate
• number of concurrent requests

When the failure threshold is reached, the breaker opens. Calls stop going to the unhealthy dependency for a short period.

Closed, Open, and Half-Open States

Circuit breakers usually have three states.

State	Meaning
Closed	calls pass through normally
Open	calls fail fast or use fallback
Half-open	a small number of test calls are allowed

Example flow:

Closed -> failures increase -> Open -> wait -> Half-open -> success -> Closed

If half-open test calls fail, the breaker opens again.

Example: A Struggling Recommendation Service

Imagine a product detail page calls recommendation-service to show related products. The feature is useful, but it is not critical to checkout. If the recommendation service is slow, the product page should still load.

Without a circuit breaker, every user request keeps waiting for the recommendation call until it times out. Under high traffic, threads, connections, and workers can pile up because a secondary feature is unhealthy.

With a circuit breaker, the flow can look like this:

Condition	Application behavior
service is healthy	show recommendations from the API
a few calls fail	record failures and keep watching
failures cross the threshold	breaker opens and stops API calls temporarily
breaker is open	show a fallback, such as popular products from cache
cooldown passes	allow a few test calls in half-open
test calls succeed	breaker closes and normal traffic resumes
test calls fail	breaker opens again

Simple implementation example:

type BreakerState = "closed" | "open" | "half-open";

class SimpleCircuitBreaker {
  private state: BreakerState = "closed";
  private failures = 0;
  private openedAt = 0;

  constructor(
    private readonly failureThreshold = 3,
    private readonly cooldownMs = 10_000,
  ) {}

  async run<T>(operation: () => Promise<T>, fallback: () => T): Promise<T> {
    if (this.state === "open") {
      const readyForTest = Date.now() - this.openedAt > this.cooldownMs;

      if (!readyForTest) {
        return fallback();
      }

      this.state = "half-open";
    }

    try {
      const result = await operation();
      this.failures = 0;
      this.state = "closed";
      return result;
    } catch (error) {
      this.failures += 1;

      if (this.state === "half-open" || this.failures >= this.failureThreshold) {
        this.state = "open";
        this.openedAt = Date.now();
      }

      return fallback();
    }
  }
}

Usage:

const recommendationBreaker = new SimpleCircuitBreaker(3, 10_000);

async function getProductRecommendations(productId: string) {
  return recommendationBreaker.run(
    () => fetchRecommendationsFromApi(productId),
    () => getPopularProductsFromCache(),
  );
}

In production, you usually do not need to write your own circuit breaker. Many stacks already have mature resilience libraries with metrics, sliding windows, concurrency limits, and event hooks. This small example is meant to show the core idea: when a dependency is unhealthy, the application stops pushing it harder and chooses a safer response.

Circuit Breaker vs Retry

Retry and circuit breaker solve different parts of resilience.

Pattern	Helps when	Risk
Retry	failure is temporary	can amplify load
Timeout	dependency is too slow	may cut off slow but valid work
Circuit breaker	dependency is unhealthy	may reject calls during recovery

Use them together carefully. Retry should have limits, backoff, and jitter. Circuit breaker should prevent retries from becoming a traffic storm.

Fallback Strategies

A fallback should be useful and honest.

Examples:

• return cached data
• show a degraded experience
• queue work for later
• skip non-critical recommendations
• return a clear error for critical actions

Not every operation should have a fallback. For payment capture, a fake success would be dangerous. For product recommendations, hiding the widget may be fine.

What to Monitor

Circuit breakers need observability.

Track:

• breaker state changes
• dependency latency
• timeout count
• fallback count
• rejected call count
• recovery success rate
• user-facing error rate

Example dashboard:

Metric	Healthy signal
open breaker count	low and short-lived
fallback rate	occasional, not constant
dependency latency	within SLO
half-open success	increasing during recovery

Common Mistakes

Retrying too much before opening the breaker

Too many retries can make the dependency worse. Keep retry budgets small.

Using the same threshold everywhere

Critical dependencies, optional dependencies, and slow batch integrations need different settings.

Returning misleading fallbacks

Fallbacks should not pretend an action succeeded if it did not.

Not exposing breaker state

If teams cannot see when a breaker opens, debugging becomes guesswork.

Checklist

• Set timeouts before adding retries.
• Use bounded retries with backoff and jitter.
• Define failure thresholds per dependency.
• Choose honest fallback behavior.
• Track breaker state changes.
• Alert on breakers that stay open too long.
• Test half-open recovery.
• Avoid fake success for critical writes.
• Document what users experience during fallback.

FAQ

Is circuit breaker only for microservices?

No. It is useful for any unreliable dependency: external APIs, databases, queues, search services, and internal services.

Should every dependency have a circuit breaker?

Not always. Use it where failure can consume resources, cause cascading issues, or hurt user experience.

Can circuit breakers replace monitoring?

No. They reduce blast radius, but you still need monitoring to understand and fix the underlying failure.

Conclusion

Circuit breakers make systems more resilient by stopping unhealthy dependencies from consuming endless resources.

They work best with timeouts, bounded retries, honest fallbacks, and strong observability. The pattern is simple, but the production behavior depends on thoughtful thresholds and clear user impact.