Prometheus & Grafana

Prometheus & Grafana

Prometheus collects and stores metrics, while Grafana visualizes them with beautiful dashboards. Together they form the most popular open-source monitoring stack.

The Monitoring Stack

  ┌─────────────────────────────────────────┐
  │            Grafana Dashboards           │
  │  ┌──────────┐  ┌──────────┐  ┌───────┐ │
  │  │ CPU/Mem  │  │ Request  │  │Errors │ │
  │  │ Graphs   │  │ Rate     │  │  Panel│ │
  │  └──────────┘  └──────────┘  └───────┘ │
  └──────────────────┬──────────────────────┘
                     │ Queries
                     ▼
  ┌─────────────────────────────────────────┐
  │           Prometheus Server             │
  │  ┌──────────────────────────────────┐   │
  │  │ Time-Series Database (TSDB)      │   │
  │  └──────────────────────────────────┘   │
  │  ┌──────────────────────────────────┐   │
  │  │ PromQL Query Engine              │   │
  │  └──────────────────────────────────┘   │
  └──────────────────┬──────────────────────┘
                     │ Scrapes
          ┌──────────┼──────────┐
          ▼          ▼          ▼
  ┌──────────┐ ┌──────────┐ ┌──────────┐
  │App Metrics│ │Node Exp. │ │MySQL Exp.│
  │  /metrics│ │:9100     │ │:9104     │
  └──────────┘ └──────────┘ └──────────┘

Prometheus Configuration

  # prometheus.yml
  global:
    scrape_interval: 15s
    evaluation_interval: 15s

  scrape_configs:
    - job_name: 'myapp'
      static_configs:
        - targets: ['app:8080']
      metrics_path: '/metrics'

    - job_name: 'node'
      static_configs:
        - targets: ['node-exporter:9100']

PromQL Examples

  # Request rate per second
  rate(http_requests_total[5m])

  # Error rate (5xx responses)
  rate(http_requests_total{status=~"5.."}[5m])

  # CPU usage percentage
  100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

  # Memory usage
  (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100