metricq

MetricQ is a highly-scalable, distributed metric data processing framework based on RabbitMQ.
This repository used to be the central repository, but has since been splitted into several other
repositories.

The different MetricQ language implementations can be found here:

• C++
• Python

The proto files of the used Protobuf definitions can be found here.

Documentation

Given the distributed architecture of MetricQ, the documentation is scattered over several repositories and webpages:

• The documentation of the Python implementation is a good entry point
• The Wiki in this repository still contains some information
• The HTA README.md talks about some details of persistence layer
• The Management protocol documentation lists all RPC’s used

There are also a lot of client implementations available:

• Tools using the Python implementation
• Examples for the Python implementation
• Tools using the C++ implementation
• Examples for the C++ implementation
• All MetricQ clients in the MetricQ GitHub organization
• MetricQ client implementations on GitHub

Setup development environment with docker-compose

Note: During the startup, especially on the first one, errors and restarts of some services are normal! Please be patient.

Just run:

docker-compose -f docker-compose-development.yml up

This will setup:

• Grafana server (port 3000 forwarded to localhost:3001)
• CouchDB server (port 5984 forwarded to localhost)
• RabbitMQ server (port 5672 and 15672 forwarded to localhost)
• MetricQ Wizard (port 3000 forwarded to localhost)
• MetricQ Webview (port 80 forwarded to localhost:3002)
• MetricQ Explorer (port 80 forwarded to localhost:3004)
• MetricQ Wizard backend (port 8000 forwarded to localhost)
• metricq-sink-websocket (port 3000 forwarded to localhost:3003)
• MetricQ Manager
• metricq-grafana (port 4000 forwarded to localhost)
• C++ example source generating a metric called dummy.source
• metricq-rabbitmq-source providing metricq.rabbitmq.[...] performance metrics for the running RabbitMQ server
• metricq-source-sysinfo providing loalhost.[...] performance metrics for the docker host
• metricq-db-hta database that stores the metrics
• metricq-example-combinator a combinator that can combine metrics into new metrics

By default, all logins are admin / admin. Do not use this dockerfile for production use!

To run it in the background append -d:

docker-compose -f docker-compose-development.yml up -d

To stop everything run:

docker-compose -f docker-compose-development.yml stop

To stop and remove everything run

docker-compose -f docker-compose-development.yml down

Connecting to the MetricQ network

You can now connect to the network with amqp://admin:admin@localhost as url and dummy.source as a metric. Using the examples from metricq-python.

pip install ".[examples]"
./examples/metricq_sink.py --server amqp://admin:admin@localhost -m dummy.source

Setup clustered development environment with docker-compose

If you follow the steps from above instead with docker-compose-cluster.yml,
three RabbitMQ nodes will be set up.
On start, they will automatically form a cluster.

The container names will be (might be different for your specific setup):

• metricq_rabbitmq-server-node0_1
• metricq_rabbitmq-server-node1_1
• metricq_rabbitmq-server-node2_1

By default, all MetricQ agents started from the compose file will connect to
rabbitmq-server, which resolves to any of the three nodes.

Note: You need to make sure to use the new BuildKit by for instance setting the ENV variable
COMPOSE_DOCKER_CLI_BUILD to 1, or use docker-compose newer than 1.28.0-rc3.

Configure like live Cluster

• Create a user-policy with
  • Name: ManagementAsHA
  • Pattern: management
  • Definition: ha-mode: all

Connecting to nodes from docker network

Use the hostname rabbitmq-server and the client will connect to random node in the cluster.

For specific nodes, use the hostnames rabbitmq-node0, rabbitmq-node1, or rabbitmq-node2.

Connecting to nodes from host or remotely

The different RabbitMQ nodes are listening on the network interface of their host.

• rabbitmq-node0: 5671 / 15671
• rabbitmq-node1: 5672 / 15672
• rabbitmq-node2: 5673 / 15673

Acknowledgements

This work is supported in part by the German Research Foundation (DFG) within the CRC 912 - HAEC.

Primary Reference

Thomas Ilsche, Daniel Hackenberg, Robert Schöne, Mario Bielert, Franz Höpfner and Wolfgang E. Nagel: MetricQ: A Scalable Infrastructure for Processing High-Resolution Time Series Data 📕 2019 IEEE/ACM Industry/University Joint International Workshop on Data-center Automation, Analytics, and Control (DAAC), Denver, CO, USA, 2019, pp. 7-12, DOI: 10.1109/DAAC49578.2019.00007.

Additional Reference

Thomas Ilsche: Energy Measurements of High Performance Computing Systems: From Instrumentation to Analysis 📕 2020 Doctoral dissertation TU Dresden, URN: de14-qucosa2-716000">urnde14-qucosa2-716000