Observability That Actually Helped During Incidents

The Before

At 2 AM, the phone rings. "Orders aren't going through."

You SSH into the server. You grep logs. You find an error — timeout connecting to the payment service. You restart the service. Orders start flowing again. You go back to sleep.

Next week, same thing. You don't know why it keeps happening. You don't know if it's getting worse. You don't know if your restart actually fixed anything or just happened to coincide with the service recovering on its own.

This was our reality before investing in observability.

Structured Logging

The first and cheapest win: structured logging. Every log line is JSON with a consistent schema:

const logger = {
  info: (message: string, context: Record<string, unknown>) => {
    console.log(JSON.stringify({
      level: 'info',
      timestamp: new Date().toISOString(),
      message,
      ...context,
    }));
  },
  error: (message: string, error: Error, context: Record<string, unknown>) => {
    console.error(JSON.stringify({
      level: 'error',
      timestamp: new Date().toISOString(),
      message,
      error: error.message,
      stack: error.stack,
      ...context,
    }));
  },
};

The key fields we standardized across every service:

• correlationId — propagates across service boundaries
• userId — who triggered this
• service — which service wrote this log
• duration — how long the operation took

Now you can trace a single order through five services by searching for its correlation ID.

The Dashboards That Mattered

We didn't build dashboards because they looked good in demos. We built them because specific incidents demanded them.

Dashboard 1: Payment Gateway Health

After the Tranglo timeout incident, we built a dashboard scoped per gateway:

┌─────────────────────────────────────────────────────────┐
│  Payment Gateway Health                        [Last 1h] │
├─────────────────────────────────────────────────────────┤
│  Paywell      ████████████████  99.7%  (1,247 req)      │
│  Ding         ████████████████  99.2%  (892 req)        │
│  Tranglo      ████████░░░░░░░░  67.3%  (634 req)  ⚠️    │
│                                                         │
│  Tranglo Latency (p95)                                  │
│  ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁███  12.4s ↑        │
│                                                         │
│  Circuit Breaker State                                  │
│  Tranglo:  ● OPEN  (since 14:32)                        │
│  Ding:     ● CLOSED                                     │
│  Paywell:  ● CLOSED                                     │
└─────────────────────────────────────────────────────────┘

With this dashboard, you can tell in 2 seconds whether the problem is one gateway or systemic.

Dashboard 2: Queue Depth

Our Field Force Management platform processed tasks through RabbitMQ. When queues backed up, agents stopped receiving assignments:

// Prometheus metric pushed from each service
const queueDepth = new Prometheus.Gauge({
  name: 'rabbitmq_queue_depth',
  help: 'Number of messages in queue',
  labelNames: ['queue', 'service'],
});
 
// Updated every 15 seconds
setInterval(async () => {
  const depths = await rabbitmq.getQueueDepths();
  for (const [queue, depth] of Object.entries(depths)) {
    queueDepth.set({ queue, service: 'ffm' }, depth);
  }
}, 15000);

The dashboard showed a real-time heatmap: green queues are healthy, yellow needs attention, red needs immediate action. Alert rules fired when any queue exceeded 1,000 messages for more than 5 minutes.

Dashboard 3: API Latency by Endpoint

Not all endpoints are equal. A 500ms GET /products is fine. A 500ms POST /orders is a problem. We broke down latency by endpoint and percentile:

Service: orders-api

Endpoint              p50    p95    p99    req/s
POST /orders          120ms  340ms  890ms  45.2
GET /orders/:id       45ms   110ms  280ms  230.1
PUT /orders/:id       80ms   220ms  450ms  12.7
POST /orders/:id/sync 210ms  890ms  2.1s   3.4  ← investigate

The POST /orders/:id/sync endpoint had a p99 of 2.1 seconds. This pointed to a slow SAP callback — something we wouldn't have caught without per-endpoint latency metrics.

Correlation IDs

The single most impactful change: every incoming request gets a correlation ID. If none is provided, we generate one. It propagates through HTTP headers, RabbitMQ message properties, and log context.

// NestJS interceptor
@Injectable()
export class CorrelationInterceptor implements NestInterceptor {
  intercept(context: ExecutionContext, next: CallHandler): Observable<any> {
    const request = context.switchToHttp().getRequest();
    const correlationId = request.headers['x-correlation-id'] || uuid();
 
    // Attach to async context
    return next.handle().pipe(
      tap(() => {
        // Inject into outbound HTTP calls
        request.headers['x-correlation-id'] = correlationId;
      })
    );
  }
}

Now when someone reports "order #12345 failed", you search logs for that order's correlation ID and see the full timeline — across API gateway, order service, payment service, and notification service.

Alerting That Doesn't Wake You Up for Nothing

We set up three tiers of alerts:

1. Info — Slack notification. "Queue depth increasing." No one wakes up.
2. Warning — Slack + email. "Error rate above 2% for 10 minutes." On-call checks during business hours.
3. Critical — PagerDuty. "Payment gateway circuit breaker open." Someone wakes up.

The rule: if you get paged at 2 AM and it's a false alarm, that alert gets downgraded before the next shift. Every page should be actionable.

What We Learned

1. Start with logs, add metrics later — Structured logging with correlation IDs solved 80% of our debugging problems. Metrics were for the remaining 20% and for proactive detection.
2. Per-endpoint metrics are worth it — Aggregate latency hides problems. POST /orders being slow doesn't mean all endpoints are slow. Break it down.
3. Dashboards should answer specific questions — "Is it down?" "Which part is down?" "Is it getting worse?" Each dashboard answers exactly one.
4. Alert on symptoms, not causes — Alert on "users can't place orders" not "CPU is high." High CPU might be fine. Failed orders are never fine.
5. Correlation IDs are non-negotiable — If you take one thing from this post, make it this. A correlation ID on every request saves more debugging hours than any monitoring tool.