Logging with ELK Stack

Centralized logging with Elasticsearch, Logstash, and Kibana

2 min read | DevOps

⭐

Want the full learning experience?

Get structured courses, certificates, projects, and instructor support with LabsICT Pro.

Explore Pro Courses

Logging with ELK Stack

The ELK Stack (Elasticsearch, Logstash, Kibana) provides centralized logging for collecting, processing, and visualizing log data from across your infrastructure.

ELK Architecture


  ┌──────────┐  ┌──────────┐  ┌──────────┐  ┌──────────┐
  │ App 1   │  │ App 2   │  │ App 3   │  │  Server  │
  │ Logs    │  │ Logs    │  │ Logs    │  │  Logs    │
  └────┬────┘  └────┬────┘  └────┬────┘  └────┬────┘
       │            │            │            │
       └────────────┼────────────┼────────────┘
                    ▼
  ┌─────────────────────────────────────────────┐
  │              Logstash                       │
  │  ┌─────────┐  ┌──────────┐  ┌───────────┐  │
  │  │  Input  │─▶│  Filter  │─▶│  Output   │  │
  │  │ (Beats) │  │ (Parse,  │  │ (ES, S3)  │  │
  │  │         │  │  Enrich) │  │           │  │
  │  └─────────┘  └──────────┘  └───────────┘  │
  └──────────────────┬──────────────────────────┘
                     ▼
  ┌─────────────────────────────────────────────┐
  │           Elasticsearch                     │
  │  Index → Store → Search → Aggregate        │
  └──────────────────┬──────────────────────────┘
                     ▼
  ┌─────────────────────────────────────────────┐
  │              Kibana                         │
  │  Dashboards │ Discover │ Visualizations     │
  └─────────────────────────────────────────────┘

Logstash Pipeline


  # logstash.conf
  input {
    beats {
      port => 5044
    }
  }

  filter {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
    geoip {
      source => "clientip"
    }
  }

  output {
    elasticsearch {
      hosts => ["http://elasticsearch:9200"]
      index => "logs-%{+YYYY.MM.dd}"
    }
  }

Lightweight Alternative: Filebeat


  # filebeat.yml
  filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    fields:
      service: myapp
      environment: production

  output.elasticsearch:
    hosts: ["http://elasticsearch:9200"]
    index: "filebeat-%{+yyyy.MM.dd}"