Day 9: Monitoring and Logging in Kubernetes

🎯 Learning Objective:¶

Understand how to set up monitoring and logging in Kubernetes to gain insights into the performance and health of your applications and the cluster.

📖 Scenario:¶

You need to monitor your Kubernetes cluster and the applications running within it to ensure everything is functioning correctly and to troubleshoot issues when they arise.

📘 Explanation:¶

Monitoring in Kubernetes involves collecting, processing, and visualizing metrics to understand the performance and health of your applications and the cluster. Logging involves collecting, storing, and analyzing logs generated by your applications and Kubernetes components.

🔑 Key Concepts:¶

Metrics:

Quantitative data that provides insights into the performance and health of your system.
Examples: CPU usage, memory usage, request latency.

Logs:

Textual data that provides detailed information about the operations and events happening in your system.
Examples: application logs, system logs, audit logs.

📄 Monitoring Tools:¶

Popular tools for monitoring Kubernetes include Prometheus and Grafana.

📑 Installing Prometheus and Grafana:¶

Install Prometheus:

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml

Install Grafana:

kubectl apply -f https://raw.githubusercontent.com/grafana/helm-charts/main/charts/grafana/templates/deployment.yaml

📄 Logging Tools:¶

Popular tools for logging in Kubernetes include Elasticsearch, Fluentd, and Kibana (EFK stack).

📑 Installing the EFK Stack:¶

Install Elasticsearch:

kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/master/config/samples/elasticsearch/elasticsearch.yaml

Install Fluentd:

kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml

Install Kibana:

kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/master/config/samples/kibana/kibana.yaml

🛠️ Steps to Set Up Monitoring and Logging:¶

Set Up Prometheus:¶

Apply the Prometheus Operator manifest:

kubectl apply -f https://raw.githubusercontent.com/prometheus-operator/prometheus-operator/master/bundle.yaml

Access the Prometheus dashboard:

kubectl port-forward svc/prometheus-operated 9090:9090

Open a web browser and navigate to http://localhost:9090.

Set Up Grafana:¶

Apply the Grafana manifest:

kubectl apply -f https://raw.githubusercontent.com/grafana/helm-charts/main/charts/grafana/templates/deployment.yaml

Access the Grafana dashboard:

kubectl port-forward svc/grafana 3000:3000

Open a web browser and navigate to http://localhost:3000.

Default login: username admin, password admin.

Set Up the EFK Stack:¶

Apply the Elasticsearch manifest:

kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/master/config/samples/elasticsearch/elasticsearch.yaml

Apply the Fluentd manifest:

kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch-rbac.yaml

Apply the Kibana manifest:

kubectl apply -f https://raw.githubusercontent.com/elastic/cloud-on-k8s/master/config/samples/kibana/kibana.yaml

Access the Kibana dashboard:

kubectl port-forward svc/kibana-kb-http 5601:5601

Open a web browser and navigate to http://localhost:5601.

🔍 Detailed Example Explanation:¶

Prometheus:

Prometheus is a monitoring system and time-series database that collects metrics from configured targets at given intervals, evaluates rule expressions, and can trigger alerts.

Grafana:

Grafana is an open-source platform for monitoring and observability. It provides charts, graphs, and alerts for the web when connected to supported data sources such as Prometheus.

Elasticsearch:

Elasticsearch is a distributed search and analytics engine used for log and event data storage and analysis.

Fluentd:

Fluentd is an open-source data collector that allows you to unify data collection and consumption for better use and understanding.

Kibana:

Kibana is an open-source data visualization dashboard for Elasticsearch. It provides search, view, and interaction capabilities with data stored in Elasticsearch indices.

💡 Benefits:¶

Visibility: Monitoring and logging provide visibility into the performance and health of your applications and cluster.
Troubleshooting: Helps identify and resolve issues quickly by analyzing metrics and logs.
Proactive Management: Enables proactive management through alerts and dashboards.

📚 Additional Concepts and Examples:¶

📑 Setting Up Alerts in Prometheus:¶

You can define alerting rules in Prometheus to trigger alerts based on certain conditions.

Example:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: example-prometheus-rule
spec:
  groups:
  - name: example
    rules:
    - alert: HighMemoryUsage
      expr: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes * 100 < 10
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "High memory usage detected"
        description: "Memory usage is above 90% for more than 5 minutes."

📑 Creating Dashboards in Grafana:¶

Grafana allows you to create custom dashboards to visualize metrics from Prometheus or other data sources.

Example:

Log in to Grafana.
Add Prometheus as a data source.
Create a new dashboard and add panels to visualize metrics.

🧪 Hands-on Activity:¶

Set Up Prometheus:

Use the provided commands to install Prometheus and access the dashboard.

Set Up Grafana:

Use the provided commands to install Grafana and access the dashboard.

Set Up the EFK Stack:

Use the provided commands to install Elasticsearch, Fluentd, and Kibana, and access the Kibana dashboard.

Create Alerts in Prometheus:

Define alerting rules in Prometheus and test them.

Create Dashboards in Grafana:

Create custom dashboards in Grafana to visualize metrics from Prometheus.

Verify and Inspect:

Use kubectl get pods, kubectl logs <pod-name>, and the Prometheus, Grafana, and Kibana dashboards to verify and inspect the monitoring and logging setup.

🤝 Engage and Reflect:¶

Understanding monitoring and logging is crucial for maintaining the performance and health of your applications and Kubernetes cluster.

💬 Engage With Us: How do you plan to use monitoring and logging in your projects? What challenges did you face while implementing them? Share your experiences and thoughts.

👉 Stay tuned for more learning opportunities and keep refining your Kubernetes knowledge to stay ahead in the ever-evolving tech landscape. Let’s continue to explore, innovate, and automation

More from Vinoth Subbiah¶

Recommended from Medium¶

[

See more recommendations

](https://medium.com/?source=post_page---read_next_recirc--a72c4be852b5---------------------------------------)