Logging with ELK Stack

Logging with ELK Stack

The ELK Stack (Elasticsearch, Logstash, Kibana) provides centralized logging for collecting, processing, and visualizing log data from across your infrastructure.

ELK Architecture

  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
  │ App 1   │  │ App 2   │  │ App 3   │  │  Server  │
  │ Logs    │  │ Logs    │  │ Logs    │  │  Logs    │
  └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘
       │            │            │            │
       └────────────┼────────────┼────────────┘
                    ▼
  ┌─────────────────────────────────────────────┐
  │              Logstash                       │
  │  ┌─────────┐  ┌──────────┐  ┌───────────┐  │
  │  │  Input  │─▶│  Filter  │─▶│  Output   │  │
  │  │ (Beats) │  │ (Parse,  │  │ (ES, S3)  │  │
  │  │         │  │  Enrich) │  │           │  │
  │  └─────────┘  └──────────┘  └───────────┘  │
  └──────────────────┬──────────────────────────┘
                     ▼
  ┌─────────────────────────────────────────────┐
  │           Elasticsearch                     │
  │  Index → Store → Search → Aggregate        │
  └──────────────────┬──────────────────────────┘
                     ▼
  ┌─────────────────────────────────────────────┐
  │              Kibana                         │
  │  Dashboards │ Discover │ Visualizations     │
  └─────────────────────────────────────────────┘

Logstash Pipeline

  # logstash.conf
  input {
    beats {
      port => 5044
    }
  }

  filter {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    geoip {
      source => "clientip"
    }
  }

  output {
    elasticsearch {
      hosts => ["http://elasticsearch:9200"]
      index => "logs-%{+YYYY.MM.dd}"
    }
  }

Lightweight Alternative: Filebeat

  # filebeat.yml
  filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    fields:
      service: myapp
      environment: production

  output.elasticsearch:
    hosts: ["http://elasticsearch:9200"]
    index: "filebeat-%{+yyyy.MM.dd}"