Distributed Tracing
In microservices architectures, a single user request may travel through dozens of services. Distributed tracing tracks requests across service boundaries to identify bottlenecks and failures.
How Tracing Works
User Request Flow:
┌───────┐ ┌───────┐ ┌───────┐ ┌───────┐
│Gateway│───▶│ Auth │───▶│ Order │───▶│Payment│
└───┬───┘ └───┬───┘ └───┬───┘ └───┬───┘
│ │ │ │
└────────────┼────────────┼────────────┘
▼
┌─────────────────────────────────────────────┐
│ Trace Timeline │
│ ├─ Gateway ─────────────────────────────┤ │
│ ├─ Auth ───────┤ │ │
│ ├─ Order ──────────┤ │ │
│ ├─Payment─┤ │ │
│ │
│ Total: 250ms | Bottleneck: Payment: 120ms│
└─────────────────────────────────────────────┘
Trace Concepts
- Trace — The entire journey of a request through the system
- Span — A single unit of work within a trace (e.g., a database query)
- Context Propagation — Passing trace IDs across service boundaries
- Sampling — Collecting a subset of traces to reduce overhead
Tracing Tools
Jaeger (by Uber):
┌──────────────────────────────────────┐
│ Jaeger Agent → Collects spans │
│ Jaeger Collector → Processes spans │
│ Jaeger Query → Search & query │
│ Jaeger UI → Visualization │
└──────────────────────────────────────┘
Zipkin (by Twitter):
┌──────────────────────────────────────┐
│ Collector → Receives trace data │
│ Storage → MySQL, ES, Cassandra │
│ API → Query interface │
│ UI → Dependency graph │
└──────────────────────────────────────┘
OpenTelemetry (CNCF Standard):
┌──────────────────────────────────────┐
│ Unified API for traces, metrics, │
│ and logs. Exports to any backend. │
└──────────────────────────────────────┘