Figure 1: MQTT with Raspi and ESP family
1. MQTT - Raspberry Pi - Mosquitto and Node-Red
After considering TCP web servers and the UDP protocol in different of my blog posts (frost guard in the greenhouse 1 + 2; Geiger counter, Mosquito ) Today we take the data exchange via MQTT under the magnifying glass. In some cases, it combines the advantages of transfer via UDP with the data integrity of a TCP connection. The latter protocol is the basis for the "Message Queuing Telemetry Transport" protocol. That is, MQTT works with a secure connection. On the other hand, the framework for the transmission of measurement data to a central server, which is called broker in this context, is similarly simple as under UDP. Each participant can send and receive at will. Stations that send data to the broker are called Publishers. Clients, which reversed data from the broker, is called subscriber or subscribers. As with UDP you can easily combine both operations on a station. So that nothing confuses, there are topic areas under which one can publish or retrieve data, the so-called topics. But unlike UDP, no two network devices can chat directly together, but all transfers always run over the broker, so the server. He collects data and distributes them again. The topics are not set on the server, but through the clients. An issue exists at the moment a client publishes a message under this topic. That makes the system flexible and easy to clean.
Figure 2: MQTT - Function
As clients, transducers or display units are used with an ESP32 or ESP8266. Figure 1 shows a typical scenario. For example, the ESP32 provides temperature and humidity values from a DHT22 and receives the information from the broker whether the heater is turned on. The ESP8266 has subscribed to the news to the temperature and the humidity and receives the new values from the broker as soon as they are available. Furthermore, the controller sends the switch state to the broker under the topic heat.
A LAN or WLAN connection is a prerequisite. Naked atmega328 compatible microcontrollers divorce in this topic alone for the reason because they can not provide wireless connection and can not be programmed under micropythone in this topic. The variety of sensors that can dock directly via ADC, I2C, UART or SPI on an ESP makes the ESP family perfectly suited with the modules based on micropythone as clients for MQTT.
As a broker, however, ESPs are too narrow-sustained, because at least one Raspberry Pi has to serve. As a broker, we set the Free Software Mosquitto. It can be installed without problems on Linux machines, including on a Raspberry Pi. The Mosquitto server collects the incoming messages released from clients under a topic and distributes them on request to the clients who subscribed to these message issues (subscriber). The differences in the type of connection show the following graphics.
Figure 3: TCP - single connections no ad hoc messages from the server
TCP builds on secured individual connections. HTTP clients can only be contacted by the server by request. The data integrity is guaranteed. Monitoring of traffic is not possible.
Figure 4: UDP - Multiple connections possible
Under UDP multiple contacts between the participants are possible. Each station can exchange data with any other spontaneous data. Shipping can be done both with broadcast to all participants, as well as targeted individual stations. The data integrity is not secured. Monitoring of data traffic is only possible if appropriate broadcast commands are included in the program of the transmitting station.
Figure 5: MQTT - Targeted message delivery on request
MQTT sets up to TCP and therefore offers secured connections. The data integrity is guaranteed. Each client can set up topics on the broker that can subscribe to other clients as desired. If a topic is subscribed, the client receives the latest data on this topic for a request for the broker of this as an answer. Non-subscribed topics are not operated by the broker to the inquiring client. Monitoring of data traffic is very simple by subscribing to a Lauscher the corresponding topics.
So that we can listen to traffic, we still need a tool that we can sign up at the broker, including as a subscriber. It is also readily possible for multiple clients to subscribe to the same topic. This is an advantage of MQTT compared to the other news protocols. We will use this possibility first in the terminal on the Raspi. Later, we will use the free software Node-Red for this purpose. The program, which is also installed on the RASPI, has a graphical interface, with the help of which can be easily built by browser-compatible interfaces that allow the measurement results to be graphically displayed by the ESP measuring knechts. Node-Red is already included in the installation circumference of the Raspberry PI OS operating system and is usually installed via the software pool. But it is also managed by the command line.
Let's summarize the hardware and software for our project.
Raspberry Pi (1b +, 2, 2b, 3, 3b) ff. Designated as Raspi
SD card 8-16GB suitable for the selected Raspi
Raspi power supply, 5V, 2.5A
Monitor (only for the establishment of the Raspi)
Keyboard (just for setting up the Raspi)
Mouse (just to set up the Raspi)
HDMI cable (only for the device of the Raspi)
Imager.exe Burning program for the SD card
Software for the Windows machine
Preparing the Raspi
Get Raspberry Pi OS
Raspberry Pi OS (formerly Raspian) is derived from Debian-Linux and available in various releases. The version used here is spread for all Raspi boards of versions 1b +, 2, 2b, 2b +, 3 and 3b.
Firmware releases with different extent are offered here for download https://www.raspberrypi.com/software/operating-systems/#raspberry-pi-os-32-bit. I work in this post with a Raspberry 2B and an image that comes from the program Imager is automatically downloaded and very easy to install with just a few clicks. It is needed for this an SD card of category 10 with 8GB upwards. On my 8GB card, 45% are now available after setting up further software.
Preparing the card is very simple. We invite you Imager Download the file by double-clicking.
Figure 6: Download Raspberry Imager
Figure 7: Install Imager
After a few seconds, the installation is completed in the C: \ Program Files (X86) \ Raspberry Pi Imager folder. We can also start the application right away.
Figure 8: Start Imager
First, an operating system must be selected -> Os choose.
I have selected the recommended system.
Figure 9: Selection of the operating system
The SD card must be in the card slot and have been recognized by the system. Now comes the most dangerous part of the whole, the selection of the storage medium. That is because the content of the memory card is deleted dangerous. So you have to be sure that no important data is stored on the card. They would be irretrievably gone after the start of the burning process!
Figure 10: Map selector - here very easy
We select the card and acknowledge the security query with Yes. The download of the firmware starts, and then the writing process starts, which can take a good half an hour upwards depending on the selected firmware.
Figure 11: System was written successfully
Next we create a blank file with the name SSH using an editor and copy them to the card. If Windows wants to format the created card, leave that not to. Now we can take the card from the reader shaft and set them on the Raspi.
For the other actions, it is favorable to temporarily connect a keyboard, a mouse and a monitor on the Raspi. Furthermore, we need a free IP address in the local network, which we can assign to the Raspi. Because we want to operate servers, it is better to assign a fixed IP address better to the Raspi. The easiest way out leads through the Raspi itself. All further steps can be done via an SSH connection from another computer. So we connect the little one with the router with the router.
Now we start the Raspi. After a few settings, we are on the Raspberry Pi OS Desktop - Next, Set Country -> Germany, Next, Change Password -> Enter your own and confirm, Next, Update software, Skip, Done.
With my mini TFT display, it was now necessary to customize the screen size.
Figure 12: Call Raspi Settings
Over HEADLESS RESOLUTION You can also directly set the resolution.
Figure 13: Raspi - Set screen
In a washing as well as the activation / control of SSH access, so that we can access the RASPI from the network. Screen, keyboard and mouse on the device will then obsolete. But - still we are not so far.
Figure 14: Activate Raspi - SSH
We finish the configuration with OK.
Most of what is still to do, we can do from the command line. For this we open a terminal with the key combination Ctrl + Alt + T. The default image contains the editor nanowhich we start with the following command.
Locate the following lines in the lower part of the file dhcpcd.conf. Remove the "#" at the beginning of the line and set the values according to your home network.
We change this job in:
With Ctrl + O Save the change and with Ctrl + X Exit the editor. Restart the Raspi now.
After logging in, we will open a terminal again and check with the following command whether our changes have been made.
The issue should look like this:
We also immediately check whether the network connection works on the internet. Ping to an external URL or IP clarifies us.
Cancellation Ctrl + C, And if we are already there, we also test the SSH access. In the terminal we enter the following command.
Figure 15: Raspi - SSH test
We have to confirm the "attack" with "yes" and then sign us with the password we have given above. We have now practically invaded our system through the housekeeping entrance without having left it before. OK - something is only on a PC. But if that worked, then it works out of another computer.
The last action we carry out from the command line is the installation of the Mosquitto Broker.
A lot of packages is loaded and installed, which takes a few minutes.
Again, a series of packages are brought, unpacked and furnished. Then we ensure that Mosquitto is started with the system for booting the system every time.
The following command reveals us the version of the program and that the Mosquitto server louds on the port 1883.
The error message that the address is already used (Error: Address Already in use) we can confidently ignore because it says us only that Mosquitto is already running.
As a preparation for the next blog post, however, we should change a small thing to the configuration of the Mosquitto-Deamon. If we are now trying to access the broker, for example, from another network device, for example to our ESP32, then we would have a rejection. This is because anonymous accesses, ie without user authentication and password, are prohibited by default. We want to change that, at least in the test phase.
So we switch to the directory on the Raspi /etc/mosquitto/conf.d and create a file there with the name anonymous.conf.
Now we enter the following two lines, save the file and stop nano.
Then we restart Mosquitto
Each file in the directory /etc/mosquitto/conf.d with the ending .conf is understood as a configuration file and at the start of Mosquitto over the file /etc/mosquitto/mosquitto.conf Call up. Here we can accommodate our extensions for configuration.
Installation of node-red
On the Raspi is now missing only the Package Node-Red. Let's start the installation. I chose the way over the command line again, you know what to do.
The installation takes a little longer again. Finally, the command behalf reports again, and we can ensure that the Node Red Deamon starts with the start of the system.
This ends the installation session on the Raspi. We restart the little ones. Now that all what to run on the Raspi is doing its service and works correctly, we can calm down the keyboard, monitor and mouse from the Raspi.
Software for the windowskist
During this we get a terminal program on the Windows machine with which we can contact the RASPI via the network. Is widespread Putty. We invite you File for our system down. The EXE file is executable without installation. We save you in any directory and put us a link to the desktop. With a few clicks, the terminal program is set up after the first start. Under Connection - X11 we switch X11 forwarding one.
Figure 16: Putty - X11 Switch on forwarding
Under session Let's enter the IP of our Raspis, the port stays at 22. Then we give the connection a speaking name and save the configuration.
Figure 17: Putty - Backing up
With open we open a terminal to Raspi. It looks similar like that we've already worked with the Raspi. Only the menu bar is missing.
Figure 18: Putty - terminal window
Let's try the Mosquitto and look at if he stands. For this purpose, we need two terminal windows, each of which emulates a Mosquitto client. So we open a second terminal.
In the first terminal, we enter the following command to act as a subscriber. So this window should receive messages from the Brocker.
Figure 19: Mosquitto - Subscriber listened
In the second window, we leave a message to the broker from a publisher.
Figure 20: Mosquitto - Publisher message is sent
With the sending of the message, this appears in the subcriber window.
Figure 21: Mosquitto - Subscriber - Message has arrived
While Mosquitto is addressed by the command line, Node-Red reports as a web interface in a browser. We also test this installation. Node-Red should have been automatically started at the last boot process. That's how this should look like when we node-red with the IP of our raspis and the Port 1880 Call up.
Figure 22: Node Red on the PC in Chrome
Now only a little thing is missing for today. The Raspi no longer has a screen. To work on the command line, we can log in via Putty. But what do we do if we want to do something with the graphic surface of the Raspi. We have activated X11 forwarding when setting up Putty. We can use that if we install an X11 server on the windowskist. The competent tool is called Xming and can SourceForge-Net be downloaded. We start the setup with double-click, ignore the message from Windows and then follow the "Next" wizard.
Figure 23: XMING - Setup
Figure 24: XMING - destination
Figure 25: XMING - Component selection
Figure 26: XMING - Select start folder
Figure 27: XMING - Define links
Figure 28: XMING - Completion
From the Windows Desktop, we start right-clicking on the XMING icon the program as an administrator. We have to allow the changes to the system, then XMING pulls back into the background. Now, if we open a PuTTY terminal, and write the following command to the command line, the Thonny IDE opens onto the Windows screen from our Raspi after a few seconds.
Due to the appendage "& Disown" we can use the terminal for other purposes. Without the addition, the terminal would be blocked.
We can also find out if Python is running on the Raspi.
See there, it's already the latest version 3.9, which welcomes us friendly with the Pythonprompt. However, it has to be said that it is not Micropython but about the full-grown version Cpython. There is therefore a number of differences compared to the ESP32 / ESP8266, which concerns the language scope, but also the hardware.
The basics for our MQTT project are created hereby. In the next post we will set up an ESP32 / ESP8266 as a MQTT client. Due to the fantastic properties of Node-Red, it will later create a light, appealing application for monitoring measuring points.
This blog post is also as PDF document accessible.