Kubernetes monitoring automation using Prometheus, BlackBox, Elastalert.
One of the main problem that we face with microservices deployment is how to configure monitoring.
With deployment being so dynamic its hard to make changes.
As you can see in the blog: @tomarv2/jenkins-shared-libraries-ab64f7acac68"">https://medium.com/@tomarv2/jenkins-shared-libraries-ab64f7acac68. I discussed how can
repos can be arranged and how monitoring jobs at scale can be configured. Most of the work of adding projects to monitoring and alerting was manual.
In this project we are trying to offload the management of monitoring and alerting. When a new project is added,
user adds/updates user_input.yaml
and it will be picked up by CICD (in this case Jenkins).
Where will the monitoring be configured:
Pre-reqs:
Note:
To create dummy data for testing:
verify if static-files
directory exists:
DIR_NAME=demo1
BASE_PATH='/Users/varun.tomar/Documents/personal_github'
mkdir -p $BASE_PATH/mauto/$DIR_NAME/demo-data/monitoring/static-files
verify if alertmanager config file exists:
mkdir -p $BASE_PATH/mauto/$DIR_NAME/demo-data/alertmanager
cp $BASE_PATH/mauto/data/demo-data/alertmanager/config.yaml $BASE_PATH/mauto/$DIR_NAME/demo-data/alertmanager/config.yaml
verify if prometheus config file exists:
mkdir -p $BASE_PATH/mauto/junk/demo-data/monitoring/
cp $BASE_PATH/mauto/data/demo-data/monitoring/config.yaml $BASE_PATH/mauto/junk/demo-data/monitoring/
verify .env file exists:
Copy the example.env
and create a .env file
Update src/automation/config.yaml
pip install mauto
muato
for available optionsTrying to make it modular or plug and play type so new components can be added easily.
I am using click
to create python cli.
Python 3.6 and above is required