Part 11: Ground Control to Major Tom

January 21, 2020 - Reading time: 44 minutes

It’s time to let our hair down and play a little now that we’re done building our cluster.  We’re now going to use Prometheus, Node Exporter, and Grafana to capture various metrics such as CPU temperature, memory and network usage, etc.  This will then be visualised into lots of pretty dials and graphs.  

 

Prometheus
Prometheus is an application used to capture metrics from various sources (or exporters).  You can download the Linux armv7 version here:  https://prometheus.io/download/.  We will install Prometheus on the node that we use for admin.makerflare.com.

Let’s create a Prometheus user:

sudo useradd –no-create-home –shell /bin/false prometheus

And also two folders which we will need later on:

sudo mkdir /etc/Prometheus
sudo mkdir /var/lib/Prometheus 

Now let’s apply the user permissions to the folders:

chown prometheus:prometheus /etc/prometheus
chown prometheus:prometheus /var/lib/prometheus

Next let’s extract the file we downloaded earlier into a temp folder:

sudo tar -xf prometheus-2.15.2.linux-armv7.tar.gz

We now need to copy some files to their correct locations:

sudo cp prometheus-2.15.2.linux-armv7/prometheus /usr/local/bin
sudo cp prometheus-2.15.2.linux-armv7/promtool /usr/local/bin

and..

sudo cp -r prometheus-2.15.2.linux-armv7/consoles /etc/prometheus
sudo cp -r prometheus-2.15.2.linux-armv7/console_libraries /etc/prometheus

We can now set the correct file ownership on what we copied over:

 sudo chown prometheus:prometheus /usr/local/bin/prometheus
sudo chown prometheus:prometheus /usr/local/bin/promtool

and…

chown -R prometheus:prometheus /etc/prometheus/consoles
chown -R prometheus:prometheus /etc/prometheus/console_libraries

Now let’s create a prometheus configuration file as follows:

sudo nano /etc/prometheus/prometheus.yml

The contents of the file should be as follows:

global:
   scrape_interval: 15s
scrape_configs:
   - job_name: 'prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

Save the file, and let’s apply permissions again:

sudo chown prometheus:prometheus /etc/prometheus/prometheus.yml

Now let’s create a service file so that we can run Prometheus as a service:

sudo nano /etc/systemd/system/prometheus.service

The contents of the file should be as follows:

 [Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target 

[Service]
User=prometheus
Group=prometheus
Type=simple

ExecStart=/usr/local/bin/prometheus \

    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

 [Install]
WantedBy=multi-user.target

We can now start the service as follows:

sudo systemctl enable prometheus
sudo systemctl start prometheus

You can also check to see that the service was started successfully:

Systemctl status prometheus 

 

Node Exporter
We can use various exporters to extract metrics from Linux and push them to Prometheus.  A good one to start with is Node Exporter.  This can be downloaded here:

https://github.com/prometheus/node_exporter/releases

The file you are looking for is the armv7 version (node_exporter-0.18.1.linux-armv7.tar.gz). 

 We need to create a user just as we did for Prometheus:

 sudo useradd –no-create-home –shell /bin/false node_exporter

 We can now extract the downloaded file:

 sudo tar -xf node_exporter-0.18.1.linux-armv7.tar.gz

 Copy the file:

 sudo cp node_exporter-0.18.1.linux-armv7/node_exporter /usr/local/bin

 And set ownership…

 sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter

 We can now create a service file:

 sudo nano /etc/systemd/system/node_exporter.service

 The contents of the file should be as follows:

 [Unit]
Description=Node Exporter
Wants=network-online.target
After=network-online.target 

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Let’s enable and start the service:

sudo systemctl enable node_exporter
sudo systemctl start node_exporter

And check the status:

systemctl status node_exporter

We can repeat this section on installing node_exporter on all our cluster nodes.  Note, we only need one instance of Prometheus.  Once done, we need to edit the Prometheus configuration file we created earlier on our admin node.  This needs to be updated to add all the nodes. 

Our file should look something like this after we add all the node_exporter targets:

global:
   scrape_interval: 30s

 scrape_configs:
    - job_name: 'prometheus'
     scrape_interval: 1m
     static_configs:
       - targets: ['172.16.36.253:9090']

   - job_name: 'nodes'
     scrape_interval: 1m
     static_configs:
       - targets: ['172.16.36.197:9100']
       - targets: ['172.16.36.199:9100']
       - targets: ['172.16.36.198:9100']
       - targets: ['172.16.36.253:9100']

 Save and restart the Prometheus service:

 sudo systemctl restart prometheus

 Finally, we need to open ports on both our admin node (ports 9090 and 9100) and all the other nodes (9100):

sudo ufw allow 9090
sudo ufw allow 9100

We can also check the status of both Prometheus and Node Exporter.  For Node exporter, browse to http://localhost:9100/metrics, and you will see a text output.  You can check this on all nodes. 

On the admin mode you can also browse to http://localhost:9090/targets.  This will give you an overall status of all nodes.

Note that we have also installed an exporter on all nodes which can capture CPU temperature.  This is called rpi_exporter and can be downloaded here:  https://github.com/lukasmalkmus/rpi_exporter/releases.  This can be installed in the same manner as Node Exporter above.  This exporter uses port 9243, so that will need to be allowed on UFW.  Furthermore, you can check the status of this exporter by browsing to http://localhost:9243/metrics on each node.

First, create the appropriate user (we can create a user called prometheus):

sudo useradd --no-create-home --shell /bin/false prometheus

Edit the service file (rpi_exporter.service):

sudo nano /rpi_exporter/rpi_exporter.service

Change the line for ExecStart to /bin/rpi_exporter

Copy the executable to the bin folder:

sudo cp /rpi_exporter/rpi_exporter /bin

Copy the service file to /etc/systemd/system:

sudo cp /rpi_exporter/rpi_exporter.service /etc/systemd/system

You will also need to edit the Prometheus config file on the admin node:

sudo nano /etc/prometheus/prometheus.yml

Add the endpoint targets for rpi_exporter as follows:

    - job_name: 'rpi_exporter 
     scrape_interval: 1m
     static_configs:
       - targets: ['172.16.36.197:9243']
       - targets: ['172.16.36.199:9243']
       - targets: ['172.16.36.198:9243']
       - targets: ['172.16.36.253:9243']

 Finally, enable and start the service:

sudo systemctl enable rpi_exporter
sudo systemctl start rpi_exporter
systemctl status rpi_exporter

 

Grafana
Text metrics are fine, however it’s a lot better to visualise our data.  Grafana is an application that will help with this. 

These steps should allow you to install Grafana successfully:

sudo apt-get install -y adduser libfontconfig1
wget https://dl.grafana.com/oss/release/grafana_6.5.3_armhf.deb
sudo dpkg -i grafana_6.5.3_armhf.deb

We can now enable and start Grafana as follows:

sudo systemctl enable grafana-server.service
sudo systemctl start grafana-server
sudo systemctl status grafana-server

The Grafana GUI can be accessed at http://localhost:3000, and the default username and password are admin/admin.  You will be asked to change this password.

We now have to configure a data source for Grafana to use.  In our case, this is the IP address of our admin server, and port 9090, as follows:  http://172.16.36.253:9090.  Save and test this data source. 

 

Grafana also has a repository of numerous dashboards created by others in the community.  These can easily be imported by navigating to Dashboards -> Manage, and then clicking the import button.  In our case we’ve used Dashboard number 1860.  Enter that into the text field and press “Load”.  Assign this dashboard a name, and then select the data source we just created from the dropdown.  Finally, click import.

Your dashboard should now populate successfully and should be accessible at http://localhost:3000.

However, we can also make Grafana available externally at https://admin.makerflare.com.  We need to configure Nginx as a reverse proxy. 

Edit the nginx default file:

sudo nano /etc/nginx/sites-available/default

and add a new section as follows:

location /grafana {
proxy_pass http://localhost:3000
}

This tells nginx to reverse proxy the path “/grafana” (or https://admin.makerflare.com/grafana) to http://localhost:3000. 

Now edit the grafana.ini file:

sudo nano /etc/grafana/granfana.ini

Un-comment the lines for “root_url” and “serve_from_sub_path” by removing the semi-colon before each line.  Modify the lines as follows:

root_url = http://dev.makerflare.com/grafana

and…

serve_from_sub_path = true

Your Grafana dashboard should now be accessible at http://admin.makerflare.com/grafana.


Recap
We’ve used Prometheus, Node Exporter, and Grafana to build a monitoring system for our Pi Cluster.  We’ve also made this accessible through the Internet by using Nginx as a reverse proxy. 

About

A playground of creativity that combines Cloudflare technology with hobbyists.