Monitor your Kubernetes cluster using Prometheus and Grafana

5 min readApr 14, 2024

Note: These steps apply to Kubernetes (k8s), K3s, Minikube, production (prod), development (dev), and all kinds of Kubernetes clusters. I am performing them on Minikube.

Launch an EC2 instance of type t2.medium and install Minikube on it. You can refer to the Minikube installation guide
You can now install Prometheus using either Helm or Operator. The Operator offers advanced capabilities, including lifecycle management controllers that enable automatic updates configuration.
In this demo, we’ll be using Helm to install Prometheus.

snap install helm --classic
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

4. Before installing anything, remember to update the Helm repository so that you can access the latest additions.

helm repo update

5. After that, install the Prometheus controller and any other required controllers. Make sure to read the provided information during installation to access the Prometheus server.

helm install prometheus prometheus-community/prometheus

6. Now, check if the Prometheus pods are running or not.

kubectl get pods

So, let’s understand what ‘kube-state-metrics’ is. The Kubernetes API server exposes a few metrics of your Kubernetes cluster by default. However, to effectively monitor your Kubernetes cluster, you need more detailed information such as pod statuses, deployment statuses, service statuses, and replica counts. To address this need, Kubernetes created ‘kube-state-metrics’. When you call its API, it provides extensive information about your Kubernetes cluster. This highlights the importance of ‘kube-state-metrics’ in Kubernetes monitoring.

7. Now, let’s check the services.

kubectl get svc

You can see that all the services are of type ‘ClusterIP’. However, I want to expose the ‘prometheus-server’ service to demonstrate what the Prometheus API server looks like. So, I’ll convert the service from ‘ClusterIP’ to ‘NodePort’ type.

kubectl expose service prometheus-server --type=NodePort --target-port=9090 --name=prometheus-server-ext

You can see a new service called ‘Prometheus-server-ext’ because I specified the service name as ‘Prometheus-server-ext’ in the command.

8. Now, you can open the Prometheus server UI on port ‘9090’ using the public IP address.

kubectl port-forward --address 0.0.0.0 service/prometheus-server-ext 9090:80

-address 0.0.0.0: This tells kubectl to make the forwarded port accessible from any IP address on your local machine (not just localhost).
service/prometheus-server-ext: This specifies the Kubernetes service you want to forward traffic to (the Prometheus server in your case).
9090:80: This defines the port mapping. Traffic on port 9090 on your local machine will be forwarded to port 80 on the Prometheus service within the cluster.

Why it worked: You exposed the Prometheus service with a NodePort (32462). This command maps port 9090 on your local machine to port 80 on the service, effectively giving you a new way to access Prometheus from your local machine at http://localhost:9090.

9. Now, let’s install Grafana using Helm charts.

helm repo add grafana https://grafana.github.io/helm-charts

helm repo update

10. Now, we are installing Grafana.

helm install grafana grafana/grafana

Indeed, it’s crucial to follow the steps carefully because you’ll need to obtain the password for Grafana.

11. Now, let’s expose this Grafana service similar to how we did with Prometheus.

kubectl expose service grafana --type=NodePort --target-port=3000 --name=grafana-ext

kubectl port-forward --address 0.0.0.0 service/grafana-ext 3000:80

12. We have successfully set up Prometheus and Grafana for Kubernetes monitoring. Next, you’ll need to create a data source in Grafana using Prometheus because Grafana is a visualization platform and data comes from Prometheus.

13. Go to the Grafana dashboard, click on “Data sources,” and then select Prometheus.