janvda / balena-node-red-mqtt-nginx-tig-stack Goto Github PK

Composite docker application with "8" containers (2x Node-RED, 2x MQTT broker, Telegraf, InfluxDb, Grafana, Nginx) for deployment on Raspberry Pi through Balena.

License: Eclipse Public License 2.0

Dockerfile 9.95% Shell 2.34% JavaScript 87.71%

balena node-red balenacloud raspberry-pi mqtt-broker telegraf influxdb grafana-dashboards tig

balena-node-red-mqtt-nginx-tig-stack's Introduction

Composite docker application with "8" containers (2x Node-RED, 2x MQTT broker, Telegraf, InfluxDb, Grafana, Nginx) deployed on Raspberry Pi through Balena.

Features

This project is actually a proof of concept to demonstrate the following features:

The ability to run many containers on a Raspberry Pi 3 Model B+ (see section 1. What).
The Built and Deployment of this multi container application using the BalenaCloud services (see section 2. How to install ...).
Monitoring the system resources of the raspberry pi using the TIG stack (see section 3. System resource monitoring ...):
That Grafana is very nice and powerful tool to create dashboards (see section 4. Grafana) and that it is easy to create or update those dashboards (see section 4.1 Updating and adding ...).
It is possible to run multiple Node-RED instances on the same device (see section 5. Node-RED).
It is possible to run multiple MQTT brokers on the same device (see section 6. MQTT brokers).
A USB memory stick connected to the pi can be used for storing specific data (in this case it is the influxdb data) (see section 7. USB memory stick).
It is possible to access the Grafana user interface and the 2 Node-RED editors via the internet (see see section 8. Internet Access).

1. What

This github repository describes a composite docker application consisting of "8" containers that can be deployed through BalenaCloud on any arm device (e.g. a Raspberry Pi 3 Model B+) running the balena OS.

So, this application consists of the following 8 docker containers (= TIG stack + 2x Node-RED + 2x MQTT broker + Nginx )

Telegraf - agent for collecting and reporting metrics and events
Influxdb - Time Series Database
Grafana - create, explore and share dashboards
2x Node-RED - flow based programming for the Internet of Things (accessible through path /nodered )
2x MQTT-broker - lightweight message broker
nginx - is open source software for web serving, reverse proxying, caching, load balancing,....

2. How to install this application on an edge device

It is very easy to install this application using the BalenaCloud services through following steps:

Balena Setup: you need a BalenaCloud account and your edge device must be running the BalenaOs. You also need to create an application in your balena dashboard and associate your edge device to it (see balena documentation).
clone this github repository (this can be done on any device where git is installed) through the following command git clone https://github.com/janvda/balena-node-red-mqtt-nginx-TIG-stack.git (instead of directly cloning the repository it migh be better to fork the github repository and then clone this forked repository).
Move into this repository by command cd balena-edge-device-monitoring
Add balena git remote endpoint by running a command like git remote add balena <USERNAME>@git.www.balena.io:<USERNAME>/<APPNAME>.git
push the repository to balena by the command git push balena master (maybe you need to add the option --force the first time you are deploying).

3. System resource monitoring using the TIG Stack (Telegraf, Influxdb & Grafana)

The system resource monitoring is realized by the TIG stack and happens as follows:

The Telegraf container collects the system resource metrics (memory, CPU, disk, network, ...) of the raspberry pi device and sends them to
the Influxdb container that will store them in the influx database.
The Grafana container has a dashboard (see screenshot below) showing these system metrics that it has retrieved from the influxdb.

3.1 Telegraf

The table below specifies the environment variables that can be set in the Balena Device Service Variables panel for the Telegraf Service. Note that the Default Value is defined in docker-compose.yml

Name	Default Value	Description
interval	`60s`	Frequency at which metrics are collected
flush_interval	`60s`	Flushing interval (should not set < `interval`)

4. Grafana

The Grafana user interface can be accessed at port 3000 of the host OS. The login and password is admin. The name of the dashboard is system metrics.

Here below a screenshot of the system metrics dashboard that is also provisioned by this application ( file is grafana\dashboards\system metrics.json)

4.1. Updating / Adding new Grafana Dashboards

If you want to add a new Grafana dashboard then this can be done through following steps (Updating an existing dashboard can be done in a similar way):

Create the new dashboard using the Grafana UI.
From the settings menu in Grafana UI select View JSON and copy the complete json file (don't use the grafana UI export feature as this will template the datasource and will not work due to that).
Save the json contents you have copied in previous step into a new file in folder grafana\dashboards with extension .json (e.g. mydashboard-02.json)
Substitute the ID number you can find in that file just after field "graphTooltip" by null. E.g. "id": 1, should be changed into "id": null,
Commit your changes in git and push them to your balena git remote endpoint (git push balena master)

5. Node-RED

The application consists of 2 Node-RED containers:

node-red: its editor is accessble through Host OS port and path : <Host OS>:1880/node-red/
node-red-test : its editor is accessble through Host OS port and path : <Host OS>:1882/node-red-test/

Note that both Node-RED editors are protected by a user name and a hashed password that must be set through the environment variables USERNAME and PASSWORD. The Node-RED security page describes how a password hash can be generated. You can set these environment variables using your Balena dashboard either under:

Application Environment Variables (E(X)) - this implies that both Node-RED instances will have the same username and password.
Service Variables (S(X))

Notes:

node-red-data and node-red-test-data are 2 named volumed used for the \data folder of respectively node-red and node-red-test. Take care that the settings.js is only copied during the initial deployment of the application. So when the application is redeployed e.g. due to changes, then the settings.js is not recopied to the \data folder. (see also How to copy a file to a named volume?)
In order to assure that nodes installed through npm install are not lost after a restart of the container, you must assure that they are installed in the \data directory. So you must first do a cd \data and then execute the npm install .... command. This will assure that the node is correctly installed under folder \data\node_modules\ and that it will persist after restarts of the container.

6. MQTT broker

This application consist of 2 Mosquitto MQTT-brokers:

mqtt which is listening to Host OS port 1883
mqtt-test which is listening to Host OS port 1884

7. Setup of the USB memory Stick for Influxdb

The data of the influxdb will be stored in the mount location \mnt\influxdb. The influxdb container is configured (see Dockerfile and my_entrypoint.sh) so that a USB drive (e.g. a USB memory stick) with label influxdb will be mounted to this mount location. It is currently also expecting (see Dockerfile) that this USB drive is formatted in ext4 format.

If no USB drive (or memory stick) with label influxdb is connected to the raspberry pi then the named volume influxdb-data will be mounted to this location as is specified in the docker-compose.yml file.

Notes

the current Balena version doesn't yet support the definition of a volume for such a mounted drive in the docker compose yaml file therefore this is handled through the influxdb container setup as described here above.
It is not possible to mount the same USB drive also in telegraf container (I have tried that) and consequently telegraf is not able to report the disk metrics for this USB drive.

8. Internet access via Balena's public URL and Nginx.

The nginx container has been configured so that when you enable the Balena public device URL that you can access the following applications over the internet:

Grafana User Interface via <public device URL>
Node-RED editor of the container node-red via <public device URL>\node-red
Node-Red dashboard UI of the container node-red via <public device URL>\node-red\ui
Node-RED editor of the container node-red-test via <public device URL>\node-red-test
Node-RED dashboard UI of the container node-red-test via <public device URL>\node-red-test\ui

In case you get a 502 Bad Gatewayerror or a message like Cannot GET /node-red-test/ when trying to access the above public URLs then this might be fixed by restarting the nginx container !

Credits

balena-node-red-mqtt-nginx-tig-stack's People

Contributors

Stargazers

Watchers

Forkers

mtoko mkinoo sandeepk17 robodock diverjoe xandfury soester epabrego 0x1d ratte bigboss21x rusyadiramli linusjun bbascbr philippefouquet nsazunic luisgcu pdguevara cbarth1530 rfellhauer20 tomknapen bobtronic ndanzi ky3orr jrtimmerman arsiesis anilkusuma jameskim-ai flo3561 stevefielding pkharsimran lachlancresswell puccaso rohanshukla94 poynting algmm hirechrisjohnston jactor2 siranosnut3407

balena-node-red-mqtt-nginx-tig-stack's Issues

Not able to secure node-red editors - node-red-admin module not installed

Tried to secure both node-red UIs but so far no luck. Looking at node-red website instructions and using their suggestion to use:
node -e "console.log(require('bcryptjs').hashSync(process.argv[1], 8));" your-password-here, but that didn't work (password apparently is not updated).
Attempted to use node-red-admin module, but found out that it is not installed. I am investigating this issue, will report back findings. BTW, great work! I am planning to use this in a project I am working on, thanks!

docker container dashboard shows the same container with different container_names

The problem is that when a container becomes restarted it gets each a container_name (in fact the suffix of the container is different).
FYI an example of the container_name of the influxdb container influxdb_643150_676670

This is a problem for the docker containers grafana dashboard as it is based on the container_name which means that it considers a restarted container or redeployed container as a new container.

One solution would be to assure that the container_name doesn't change during restarts, but I have tried using the container_name field in the docker compose yaml without success.

system metrics dashboard reports an average user CPU usage of 35%

When checking the CPU usage in the system metrics dashboard, you can see that it reports an average user CPU usage of 35% is reported although besides the telegraf monitoring there is not much running.

This problem got introduced at 2018-11-23 somewhere between 22:26 and 22:31.
Most likely this is the period when we enabled docker monitoring.

how do you send data to the two mosquitto broker ??

hello
i have a web server that have nginx and 2 docker-compose that have each 1mqqt 1 grafana 1 node red
i want to know how did you manage mqtt sub and hub i cant send data to any mqtt broker from the outside any idea

Influxdb stuck in start loop

The symptoms are similar to what @janvda experienced here however, the ports are in the right order.

I've followed the steps to the point, I'm not using a USB drive and I'm running it on a RPi3

I've experienced the same issues with other builds involving influxdb. I have no idea where to start, as I'm really new to balena and have very limited experience with influxdb.

Amonst the errors presented, I found these as somewhat noteworthy;

May 22 10:47:57 3fc71e2 balenad[795]: /usr/bin/entry.sh: line 29: /etc/hostname: Permission denied
May 22 10:47:57 3fc71e2 balenad[795]: /usr/bin/entry.sh: line 30: /etc/hosts: Permission denied
May 22 10:47:57 3fc71e2 balenad[795]: hostname: you must be root to change the host name
May 22 10:47:57 3fc71e2 balenad[795]: mount: only root can use "--types" option
May 22 10:47:57 3fc71e2 balenad[795]: mount: only root can use "--move" option
May 22 10:47:57 3fc71e2 balenad[795]: mount: only root can use "--move" option
May 22 10:47:57 3fc71e2 balenad[795]: mount: only root can use "--move" option
May 22 10:47:57 3fc71e2 balenad[795]: time="2019-05-22T10:47:57.174124190Z" level=warning msg="unknown container" container=49c6f30a0d49f00bbe9546ce4b64194fe5c4828165174431a22721f33cb55699 module=libcontainerd namespace=plugins.moby
May 22 10:47:57 3fc71e2 balenad[795]: mount: only root can use "--move" option
May 22 10:47:57 3fc71e2 balenad[795]: umount: /dev: umount failed: Operation not permitted
May 22 10:47:57 3fc71e2 balenad[795]: mount: only root can use "--move" option
May 22 10:47:57 3fc71e2 balenad[795]: ln: cannot remove ‘/dev/ptmx’: Permission denied
May 22 10:47:57 3fc71e2 balenad[795]: mount: only root can use "--types" option
May 22 10:47:57 3fc71e2 balenad[795]: /sbin/udevd
May 22 10:47:57 3fc71e2 balenad[795]: mount: can't find LABEL=influxdb
May 22 10:47:57 3fc71e2 balenad[795]: Event: Service started {"service":{"appId":1465407,"serviceId":263421,"serviceName":"node-red","releaseId":929996}}
May 22 10:47:57 3fc71e2 balenad[795]: Event: Service started {"service":{"appId":1465407,"serviceId":262989,"serviceName":"influxdb","releaseId":929996}}
May 22 10:47:58 3fc71e2 balenad[795]: Warning: Ignoring unsupported or unknown compose fields: containerName

node-red nodes installed through `npm install` are lost when restarting/redeploying the node-red containers

E.g. in a Node-RED container I have installed node-red-contrib-math by entering the following command
npm install bartbutenaers/node-red-contrib-math

This did successfully install the math node but when I restarted the container, I noticed that this node was no longer installed.

"502 Bad Gateway" or "Cannot GET /node-red-test/"

Sometimes we can no longer access the Grafana user interface or the Node-RED editor through the public device URL after a deployment (git push resin master). In that case we get a 502 Bad Gateway or Cannot GET /node-red-test/

Note that in most cases this error can be fixed by restarting the nginx container !

mounting USB memory stick in influxdb container didn't work.

Using the command df``in the influxdb container I have noticed that the USB drive is not mounted. I tried to manually mount it using the command mount \mnt\usbdrivebut it reported following error:root@98d476aa8909:/# mount /mnt/usbdrive
mount: /dev/sda1 is already mounted or /mnt/usbdrive busy
root@98d476aa8909:/#`

The problem is most likely caused due to the fact that same device (/dev/sda1 = my USB memory stick) is already mounted in the telegraf container and apparently you can not mount the same device twice.

Node-RED readonly

When I want to change backlight of RPi display, it trhrows:
Error: EROFS: read-only file system, open '/sys/class/backlight/rpi_backlight/bl_power

Is there a way to do this on readonly FS or how do I change the FS to writable?

Thank you in advance.

problems when redeploying node-red services with latest docker image.

There is currently an issue with latest nodered/node-red-docker:rpi-v8 docker image which might give problems.

This problem typically appears when you have recently redeployed this application because the balena build machine will detect that there is new build version available and build against that version.

Moreover when going back to previous version = nodered/node-red-docker:0.20.7-rpi-v8 you might encounter issues when using latest node-red-dashboard version.
For more information see https://discourse.nodered.org/t/node-red-dashboard-ui-button-syntaxerror-unexpected-token/15844 (it also mentions a solution).

make location where influxdb data is stored configurable.

Currently the composite docker application assumes that a USB memory stick with label influxdb in ext4 format is connected to the raspberry pi.

The idea is to make this more configurable so that it also works when we don't have a USB memory stick.
The objective is still that the data should not be lost when restarting the influxdb container.

Missing "$" in telegraf.conf

I believe that there is a missing "$" in the telegraf.conf file on line 53.

flush_interval = "flush_interval"

It seems it should be
flush_interval = "$flush_interval"

I discovered this when I had been using a fork from your project and then pulled in your changes. The build no longer started Telegraf properly. I added the $ and it started working perfectly again.

BTW - this is incredibly amazing!!!!!

docker compose is using "image: tobi312/rpi-nginx" while it should be "build: nginx".

This problem is already fixed in the node-red branch.
It still must be fixed in the master branch.

Monitoring the Node-RED application using "appmetrics"

The goal of this feature request is to specifically monitor the node-red application running in the node-red containers using Telegraf and Grafana.

node-red install error: sudo: no tty present and no askpass program specified

I am receiving the following errors from node-red and node-red-test installation scripts

[node-red-test]  sudo: no tty present and no askpass program specified
[mqtt]           [=========================================================] 100[node-red-test]
[mqtt]           [=========================================================] 100[node-red]       Removing intermediate container c3783464b4e2
[mqtt]           [=========================================================] 100[node-red]       The command '/bin/sh -c sudo npm install -g --unsafe-perm node-red-admin' returned a non-zero code: 1
[mqtt]           [=========================================================] 100[node-red-test]  Removing intermediate container 5dd7c9e2425b
[mqtt]           [=========================================================] 100[node-red-test]  The command '/bin/sh -c sudo npm install -g --unsafe-perm node-red-admin' returned a non-zero code: 1

Container in endless installing - starting - installing - stopping loop since upgrade from Resin Os 2.15.1+rev2 to balenaOS 2.27.0+rev1

see : https://forums.balena.io/t/container-in-endless-installing-starting-installing-stopping-loop-since-upgrade-from-resin-os-2-15-1-rev2-to-balenaos-2-27-0-rev1/4790

move the influxdb data to an external drive (connected via USB)

Currently all the influxdb data is written to a "named volume". This means that we have the advantage that the data is not lost when the application becomes redeployed but it also means that the data is still written to the SD card. I don't think it is a good practice to do a lot of writing to an SD card. This increases the risk that the SD card got corrupted. So instead of writing the influxdb data to an SD card, I think it is better to write this to an external drive that is connected to the USB port of the raspberry pi.