ELK-stack

ELK Stack

• Monolithic application -used traditional
  • there will be only one whole app, which will log to any DBs / any storage. If any issue comes, developer will check log and rectify it and move it on production
• Micro service Architecture - used Now
  • there will be lot of Micro services, i.e., For Ex: consider shopping app
    • authentication , payment , shipping delivery will be as different services and will create logs separately. in addition to some maybe scalable as well
  • Logs will be generated by all services
  • ELK is a distributed log Management

• Monolithic & Micro service architecture diagram

---

ELK COMPONENTS

• LOGSTASH - Data aggression and Processing
• ELASTIC SEARCH - Indexing & Storage
• KIBANA - Analysis & visualisation

-Complex / Real time architecture of ELK STACK

• BEATS (Data Collection) --> KAFKA / RABBITMQ / REDIS (Buffering or Caching) --> LOGSTASH --> ELASTIC SEARCH --> KIBANA

---

Installing ELK Stack

---

JAVA and NGINX are pre-requisites

---

Installing Elastic search

• wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
• For debian we need to install all-transport-http package - sudo apt-get update sudo apt-get install apt-transport-https
• Adding repository
  • Open vi /etc/apt/source.list
  • Add the following lines
    • deb http://extras.ubuntu.com/ubuntu trusty main
    • deb-src http://extras.ubuntu.com/ubuntu trusty main
    • deb http://packages.elastic.co/elasticsearch/2.x/debian stable main
• Now Install elastic search - apt-get install elasticsearch
• After installed, change the following configs on vim /etc/elasticsearch/elasticsearch.yml ,
  • network.host: 0.0.0.0
  • discovery.type: single-node

  You can't use your public IP in network.host

• IMPORTANT - If you are using AWS instances, you need to check the successful running of elaseticsearch you can run curl http://public-IP:9200 on terminal
• getting the following output shows everything is fine
  • { "name" : "ip-172-31-28-103", "cluster_name" : "elasticsearch", "cluster_uuid" : "_na_", "version" : { "number" : "7.12.1", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "3186837139b9c6b6d23c3200870651f10d3343b7", "build_date" : "2021-04-20T20:56:39.040728659Z", "build_snapshot" : false, "lucene_version" : "8.8.0", "minimum_wire_compatibility_version" : "6.8.0", "minimum_index_compatibility_version" : "6.0.0-beta1" }, "tagline" : "You Know, for Search" }

• Or in Browser try running http://public-IP:9200

---

Installing Logstash

• Run sudo apt-get install logstash

Installing Kibana

• Run sudo apt-get install kibana
• sudo service kibana start
• then change the config file mentioned in code - kibana.yml

---

Special Things to Note on ELK

1. Install Beats to collect files / metrics / health / packet of a server - Ex: apt-get install metricbeat
2. Make sure elasticsearch.yml, kibana.yml, filebeat.yml & metricbeat.yml are configured for your requirement
3. you can set password for elasticsearch by going to /usr/share/elasticsearch location and running bin/elasticsearch-setup-passwords interactive
   1. bin/logstash --config.test_and_exit -f <path_to_config_file> - use the following command to check .conf is perfectly fine

Before changing you need to make sure xpack.security.enabled: true is added to elasticsearch.yml

1. default user - elastic
2. After changing you need to add username and password to **`kibana.yml`**

4. You need to change respective beats .yml file to collect respective details and display on Kibana
5. Use filebeat modules enable system to make default monitoring enable for filebeat

• filebeat modules list to get list of defaults it will monitor