The Circuit Breaker Pattern in Production

The Problem

At Dvive, we integrated three payment gateways — Paywell, Ding, and Tranglo — into a single processing backend handling 10,000+ transactions daily. Each gateway had its own availability characteristics and failure modes.

The architecture was straightforward: a request came in, we routed it to the appropriate gateway, and returned the result. Until one day, Tranglo started timing out.

Not failing fast. Not returning errors. Timing out — 30 seconds per request.

Our thread pool filled up. Requests that should have gone to Paywell (which was perfectly healthy) started queuing behind Tranglo's timeouts. A single degraded dependency took down the entire payment system.

Why Retry Alone Doesn't Work

The instinct is to add retries. But retries without a circuit breaker make things worse:

// ❌ This makes the problem worse
async function processPayment(payment: Payment, gateway: Gateway) {
  try {
    return await gateway.charge(payment);
  } catch (error) {
    // Retry immediately — now we're hammering an already-degraded service
    return await gateway.charge(payment);
  }
}

Every retry adds more load to an already struggling service. It's like repeatedly calling someone who's not answering — you're not helping.

The Circuit Breaker

The pattern is inspired by electrical circuit breakers: when failures exceed a threshold, the breaker opens and requests fail immediately instead of waiting for timeouts.

class CircuitBreaker {
  private state: "CLOSED" | "OPEN" | "HALF_OPEN" = "CLOSED";
  private failureCount = 0;
  private lastFailureTime = 0;
  private successCount = 0;
 
  constructor(
    private failureThreshold: number = 5,
    private resetTimeout: number = 30000,
    private halfOpenMaxRequests: number = 3
  ) {}
 
  async execute<T>(fn: () => Promise<T>, fallback: () => T): Promise<T> {
    if (this.state === "OPEN") {
      if (Date.now() - this.lastFailureTime > this.resetTimeout) {
        this.state = "HALF_OPEN";
        this.successCount = 0;
      } else {
        return fallback(); // Fail fast — don't even try
      }
    }
 
    try {
      const result = await fn();
      this.onSuccess();
      return result;
    } catch (error) {
      this.onFailure();
      return fallback();
    }
  }
 
  private onSuccess() {
    this.failureCount = 0;
    if (this.state === "HALF_OPEN") {
      this.successCount++;
      if (this.successCount >= this.halfOpenMaxRequests) {
        this.state = "CLOSED"; // Service is healthy again
      }
    }
  }
 
  private onFailure() {
    this.failureCount++;
    this.lastFailureTime = Date.now();
    if (this.failureCount >= this.failureThreshold) {
      this.state = "OPEN"; // Stop calling the service
    }
  }
}

Gateway-Specific Configurations

Different gateways had different failure characteristics. Paywell was rock-solid but slow. Tranglo was fast but had periodic brownouts. Ding was somewhere in the middle.

We tuned the breaker per gateway:

const gatewayConfigs = {
  paywell: {
    failureThreshold: 3,
    resetTimeout: 60000,    // 1 minute cooldown
    requestTimeout: 5000,   // 5s timeout
  },
  tranglo: {
    failureThreshold: 2,    // More aggressive — Tranglo degraded often
    resetTimeout: 120000,   // 2 minute cooldown
    requestTimeout: 3000,   // 3s timeout
  },
  ding: {
    failureThreshold: 5,
    resetTimeout: 30000,    // 30s cooldown
    requestTimeout: 8000,
  },
};

Observability

The circuit breaker is useless if you don't know when it's open. We added Prometheus metrics:

const circuitBreakerState = new Prometheus.Gauge({
  name: "circuit_breaker_state",
  help: "State of circuit breaker (0=closed, 1=open, 2=half_open)",
  labelNames: ["gateway"],
});
 
const circuitBreakerFailures = new Prometheus.Counter({
  name: "circuit_breaker_failures_total",
  help: "Total number of circuit breaker failures",
  labelNames: ["gateway"],
});

Grafana dashboards showed breaker state per gateway in real-time. When Tranglo's breaker went red, the on-call engineer knew immediately — no need to dig through logs.

What I'd Do Differently

1. Bulkhead pattern — Each gateway should have its own thread pool. A Tranglo timeout shouldn't block Paywell requests, even without a circuit breaker.
2. Request hedging — For critical payments, send the request to two gateways simultaneously and use whichever responds first.
3. SLA-based thresholds — Instead of fixed failure counts, use error rate over a sliding window. Five failures in five minutes is a problem. Five failures in five hours isn't.

Key Takeaway

A circuit breaker isn't about making your system faster. It's about making your system resilient. It trades availability of a degraded dependency for availability of the overall system.

The next time a dependency starts timing out, your system should degrade gracefully — not cascade into a full outage.