Prometheus/Grafana for Kubernetes

Prometheus/Grafana from Helm charts to monitor K8s cluster:

Note:

This guide asumes that you have a running K8s cluster and helm installed (We’ll be using helm’s charts)

• I recommend instances type similar to t2.medium or bigger

First steps:

Installing Grafana:

Install from latest stable Helm chart:

helm install stable/grafana --name grafana-testing

I choose the helm release name “grafana-testing”

Pay attention, or better, copy the final lines from the output starting at “Notes: “ You will need it.

Installing Prometheus:

Install from latest stable Helm chart:

helm install stable/prometheus --name prometheus-testing

I choose the helm release name “prometheus-testing”

Pay attention, or better, copy the final lines from the output starting at “Notes: “ You will need it.

Check your helm releases:

helm ls

Custom your Prometheus/Grafana set up:

I access both GUIs by creating a simple NodePort K8s Service (I’m testing, I don’t need/want an ELB and the costs that implies):

Exposing Prometheus via NodePort:

kubectl -f https://raw.githubusercontent.com/pulpbill/Prometheus/master/prometheus-service.yaml apply

Exposing Grafana via NodePort:

kubectl -f https://raw.githubusercontent.com/pulpbill/Prometheus/master/grafana-service.yaml apply

Check your deployed Kubernetes services (I use default namespace):

kubectl get svc -n default

Allow your IP at the security group of your K8s node so you can access that node IP at the port you’ve just created: Grafana -> yourec2ip:30003 and Prometheus -> yourec2ip:30000

Get your grafana “admin” password:

Time to use the output I mentioned before:

kubectl get secret --namespace default grafana-testing -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

Once logged in at Grafana:

• Click “Add data source” button.
• Name whatever you want, I use “Kubernetes” since it’s the thing we’re monitoring
• Choose “Prometheus” at Type
• At HTTP section, use your prometheus-server service Endpoint IP:PORT (mine is 100.96.1.9:9090)
  • You can get that by running:

    kubectl describe svc prometheus-testing-server | grep Endpoints

• At HTTP section, use “Server (Default)” for “Access” tab.
• Save & Test

Now, let’s add a Dashboard

• Go to the plus sing at the left panel and choose “Import”
• You can go pick one at : https://grafana.com/dashboards?dataSource=prometheus&search=Kubernetes
• You just add the ID of a Dashboard right below “Grafana.com Dashaboard”
  • IE: https://grafana.com/dashboards/3119
  • You would add 3119 and hit “Load”
  • At the next screen, you only must specify a “Datasource” for “Prometheus” (we named it “Kubernetes” here) and hit “Import”

You can always create your own Grafana Dashboards or customize an existing one.

Important:

While picking a Grafana Dashboard, check for “Dependencies:” at the right. Let’s say one it’s build for InfluxDB stats, that won’t work for this case.
Recommended Dashboards: 3119 - 315 - 6663 - 1621