Working with the Cayenne Dashboard - LoRa to MQTT Gateway (Part 3)

After several failed attempts to build a universal LoRa gateway with the ESP32, the following proprietary gateway was created, which allows LoRa-based IoT devices to connect to the Cayenne dashboard via MQTT. In an advanced expansion stage, the gateway will also manage IoT devices based on the ESP-Now protocol.

For the gateway, we only need one ESP32 with LoRa and OLED Displayno further components are required. Power can be supplied with any USB power supply.

Description:

The gateway can manage 32 devices with up to 8 channels each. Data is transmitted asynchronously. When the gateway receives a LoRa data packet, the first six bytes are interpreted as the device ID (typical mac address). The gateway then checks a device list to see if this device is already registered. If this is not the case, the device id is saved and displayed via the web interface for registration. 

If the device is already registered, the data is read and stored per channel in a message buffer. The channel number is determined from the device number (index in the device list) * 8 + channel number of the received data. Device 0 thus has channels 0 to 7, device 1 channels 8 to 15 etc. 

A record starts with one byte for the channel number followed by one byte of the type, followed by 1 to 6 data bytes, depending on the type. After all data has been saved to the message buffer and marked as new, the response packet is compiled to the Lora device. It starts again with the six bytes device id. The message buffer then checks the channels for this device to see if there is an output data packet from Cayenne to the device that is marked new. If such a data packet is found, it is attached to the response packet, again using the relative channel number 0 to 7. As a final step, the response packet is transferred to the LoRa device.

The Cayenne.loop(1) function communicates with the IoT server. In the callback function CAYENNE_OUT_DEFAULT() all channels that are marked with new and contain an input data packet for Cayenne are searched in the message buffer. These packets are now converted and sent to the IoT server, depending on the type. After successful transfer, the New flag is reset.

A second callback function CAYENNE_IN_DEFAULT() is called whenever Cayenne has data for an action channel. The data from the IoT server is converted depending on the type and stored in the message buffer. They are marked with New so that they are sent to the device the next time LoRa communicates with the response packet.

Display:

In addition to the name, the display shows the current date and time. This includes the IP address.
The line MQTT: shows the number of successful transfers to the IoT server, LoRa the successful transfers to LoRa devices and NOW the number of successful transfers to ESP-Now devices. The latter is currently always 0 because this function is not yet implemented.

Device list and registration:

The device list is stored as a CSV file in the Flash File System (SPIFFS) so that it is preserved even if power is missing. This list can be maintained via the web server built into the gateway. Devices can be deleted and new devices can be registered.

Packets:

For the type, however, the codes are subtracted according to the IPSO Alliance Smart Objects Guidelines, but 3200 is subtracted.

Sketch:

The gateway registration with Cayenne is done in the same way as described in Part 1 of this blog series. The access data must then be entered into the sketch (yellow marking) as well as the access data for logging in to the local WLAN. The Cayenne Dashboard will not display any widgets because no device has been connected to the gateway yet.

Board for Arduino IDE = TTGO LoRa32-OLED V1

/* The MQTT Gateway forms an interface between LoRa devices or ESP Nowe devices 
 * and Cayenne MQTT dashboards. It runs on ESP32 with LoRa and OLED display
 * The configuration is done by the browser
 */
#include <Spi.H>
#include <Lora.H>
#include "SSD1306.h"
#include<Arduino.H>
#include <CayenneMQTTESP32.H>
#include <CayenneLPP.H>
#include <Wifi.H>
#include <Web.H>
#include <Time.H>
#include "FS.h"
#include "SPIFFS.h"


We get the data for this setting from the Cayenne Dashboard
#define MQTT_USER ""
#define MQTT_PASSWORD ""
#define MQTT_CLIENTID ""

Access data for local Wi-Fi
#define WIFI_SSID ""
#define WIFI_PASSWORD ""

NTP Server for time synchronization
#define NTP_SERVER "de.pool.ntp.org"
#define GMT_OFFSET_SEC 3600
#define DAYLIGHT_OFFSET_SEC 0

Pins for the LoRa Chip
#define Ss      18
#define Rst     14
#define DI0     26
Frequency for the LoRa chip
#define Band    433175000

//
#define MAXCHANNELS 256 maximum number of channels managed
#define MAXDEVICE 32 maximum number of managed devices MAXCHANNELS/MAXDEVICE = 8 results in the maximum number of channels per device

Format Flash Filesystem if not already done
#define FORMAT_SPIFFS_IF_FAILED True

#define Debug 1

Building blocks for the web server
Const Char HTML_HEADER[] =
"<! DOCTYPE HTML>"
"<html>"
"<head>"
"<meta name = "viewport" content = "width = device-width, initial-scale = 1.0, maximum-scale = 1.0, user-scalable=0>">"
"<meta http-equiv="content-type" content="text/html; charset=UTF-8">"
"<title>MQTT Gateway</title>"
"<script language="javascript">"
"function reload()
"document.location="http://%s";"
"</script>"
"<style>"
"body - background-color: #d2f3eb; font-family: Arial, Helvetica, Sans-Serif; Color: #000000;font-size:12pt; }"
"th background-color: #b6c0db; color: #050ed2;font-weight:lighter;font-size:10pt;
"table, th, td "border: 1px solid black;"
".title .font-size:18pt;font-weight:bold;text-align:center;
"</style>"
"</head>"
"<body><div style='margin-left:30px;' >";
Const Char HTML_END[] =
"</div><script language="javascript">setTimeout(reload, 10000);</script></body>"
"</html>";
Const Char HTML_TAB_GERAETE[] =
"<table style=""width:100%"><tr><th style="width:20%"">ID</th><th style="width:10%">No.</th>"
"<th style=""width:20%">Channels</th><th style="width:20%">Name</th>"
"<th style=""width:20%">Recent Data</th><th style="width:10%">Action</th></tr>";
Const Char HTML_TAB_END[] =
"</table>";
Const Char HTML_NEWDEVICE[] =
"<div style=""margin-top:20px;">%s Name: <input type="text" style="width:200px" name="devname"" maxlength="10" value=""> <button name="register" value="%s">Register</button></div>";
Const Char HTML_TAB_ZEILE[] =
"<tr><td>%s</td><td>%i</td><td>%i to %i</td><td>%s<//td><td>%s</td><td><button name="delete" value="%i">Delete</button></td></tr>";

Structures
News Buffer
Struct MSG_BUF {   uint8_t Type;   uint8_t New;   uint8_t Data[10];
};

Device definition
Struct Device {   uint8_t Active;   uint8_t Service; 0=LoRa, 1=ESP-Now   uint8_t Id[6];   String Name;   String Last;
};

Global variable
Web server instance
Web Server(80);

OLED Display
SSD1306  Display(0x3c, 4, 15);

Buffer for caching messages per channel
MSG_BUF messages[MAXCHANNELS];

List of defined devices
Device Devices[MAXDEVICE];

Id of an unregistered device
uint8_t Unknown[6];
Flag always true when a new device is detected
Boolean newGeraet = False;
Type of new device 0=LöRa 1 =ESPNow
uint8_t newGeraetType = 0;

Counters and activities Status for the display
uint32_t loraCnt = 0; Number of LoRa messages received
String loraLast = ""; Date and time of last received LoRa message
uint32_t nowCnt = 0; Number of ESP Now messages received
String nowLast = ""; Date and time of last received LoRa message
uint32_t cayCnt = 0; Number of MQTT messages sent
String cayLast = ""; Date and time of last sent MQTT message


Function returns date and time in the format yyyy-mm-dd hh:mm:ss as a string
String getLocalTime()
{   Char sttime[20] = "";   Struct Tm timeinfo;   If(!getLocalTime(&timeinfo)){     Serial.println("Failed to obtain time");     Return sttime;   }   Strftime(sttime, Sizeof(sttime), "%Y-%m-%d %H:%M:%S", &timeinfo);   Return sttime;
}

Funktion liefert eine 6-Byte Geräte-Id im format xx:xx:xx:xx:xx:xx als String
String Getid(uint8_t Id[6])
{   String stid;   Char Tmp[4];   Sprintf(Tmp,"%02x",Id[0]);   stid=Tmp;   for (uint8_t J = 1; J<6; J++) {     Sprintf(Tmp,":%02x",Id[J]);     stid = stid += Tmp ;   }   Return stid;
}

prepares the message buffer
sets all messages on done
Void initMessageBuffer() {   for (Int Ⅰ = 0;Ⅰ<MAXCHANNELS;Ⅰ++) messages[Ⅰ].New = 0;
}

Function to save the configuration
Void writeConfiguration(Const Char *Fn) {   File Q = SPIFFS.Open(Fn, FILE_WRITE);   If (!Q) {     Serial.println(Q("ERROR: SPIFFS Can't Save Configuration"));     Return;   }   for (uint8_t Ⅰ = 0; Ⅰ<MAXDEVICE; Ⅰ++) {     Q.Print(Devices[Ⅰ].Active);Q.Print(",");     Q.Print(Devices[Ⅰ].Service);Q.Print(",");     Q.Print(Getid(Devices[Ⅰ].Id));Q.Print(",");     Q.Print(Devices[Ⅰ].Name);Q.Print(",");     Q.println(Devices[Ⅰ].Last);   }
}

Function to register a new device
Void geraetRegister() {   uint8_t Ⅰ = 0;   search free entry   while ((Ⅰ<MAXDEVICE) && Devices[Ⅰ].Active) Ⅰ++;   there is no new entry we do nothing   If (Ⅰ < MAXDEVICE) {     otherwise register geraet name = entered name      or unknown if none has been entered     If (Server.hasArg("devname")) {       Devices[Ⅰ].Name = Server.Bad("devname");     } else {       Devices[Ⅰ].Name = "unknown";     }     for (uint8_t J = 0; J<6; J++) Devices[Ⅰ].Id[J]=Unknown[J];     Devices[Ⅰ].Active = 1;     Devices[Ⅰ].Service= newGeraetType;     Devices[Ⅰ].Last = "";     writeConfiguration("/configuration.csv");     newGeraet = False;   }
}

Service function of the web server
Void handleRoot() {   Char htmlbuf[1024];   Char tmp1[20];   Char tmp2[20];   Char tmp3[20];   Int Index;   was the delete button clicked ?   If (Server.hasArg("delete")) {     Index = Server.Bad("delete").toInt();
#ifdef DEGUG     Serial.Printf("Delete device %i = ",Index);     Serial.println(Devices[Index].Name);
#endif     Devices[Index].Active=0;     writeConfiguration("/configuration.csv");   }   has the Register button been clicked ?   If (Server.hasArg("register")) {     geraetRegister();   }   Send current HTML page to browser   Server.setContentLength(CONTENT_LENGTH_UNKNOWN);   Header   Wifi.localIP().Tostring().Tochararray(tmp1,20);   Sprintf(htmlbuf,HTML_HEADER,tmp1);   Server.send(200, "text/html",htmlbuf);   Form Beginning   Server.sendContent("<div class="title">MQTT - Gateway</div><form>");   Table of active devices   Server.sendContent(HTML_TAB_GERAETE);   for (uint8_t Ⅰ = 0; Ⅰ<MAXDEVICE; Ⅰ++) {      If (Devices[Ⅰ].Active == 1) {        Getid(Devices[Ⅰ].Id).Tochararray(tmp1,20);       Devices[Ⅰ].Name.Tochararray(tmp2,20);       Devices[Ⅰ].Last.Tochararray(tmp3,20);       Sprintf(htmlbuf,HTML_TAB_ZEILE,tmp1,Ⅰ,Ⅰ*8,Ⅰ*8+7,tmp2,tmp3,Ⅰ);       Server.sendContent(htmlbuf);     }   }   Server.sendContent(HTML_TAB_END);   If a new device is found, its ID and an input field for the name of the   and a button to register the new device is displayed   If (newGeraet) {     Getid(Unknown).Tochararray(tmp1,20);     Sprintf(htmlbuf,HTML_NEWDEVICE,tmp1,tmp1);     Server.sendContent(htmlbuf);   }   Server.sendContent(HTML_END);
}

Function to find a device in the device list
Return index of device or -1 if it was not found
Int findDevice(uint8_t Dev[6]) {   uint8_t J;   uint8_t Ⅰ = 0;   Boolean Found = False;   Thu {     J = 0;     If (Devices[Ⅰ].Active == 0) {       Ⅰ++;     } else {       while ((J < 6) && (Dev[J] == Devices[Ⅰ].Id[J])) {J++;}       Found = (J == 6);       If (!Found) Ⅰ++;      }    } while ((Ⅰ<MAXDEVICE) && (!Found));   If (Found) {Return Ⅰ;} else {Return -1;}
}

Function to display the status on the OLED display
Void Display() {   Display.Clear();   Display.Drawstring(0,0,"MQTT Gateway");   Display.Drawstring(0,10,getLocalTime());   Display.Drawstring(0,20,Wifi.localIP().Tostring());   Display.Drawstring(0,34,"MQTT: ");   Display.Drawstring(60,34,String(cayCnt));   Display.Drawstring(0,44,"LoRa: ");   Display.Drawstring(60,44,String(loraCnt));   Display.Drawstring(0,54,"NOW: ");   Display.Drawstring(60,54,String(nowCnt));   Display.Display();
}


Process a message from a LoRa client
Void readLoRa() {   Int devnr;   uint8_t Daniel[6];   uint8_t Channel;   uint8_t Type;   uint8_t Len;   uint8_t Dat;   Boolean Output;   Get data if available   Int packetSize = Lora.parsePacket();   have we received data ?   If (packetSize > 5) {
#ifdef Debug         Serial.println(getLocalTime());     Serial.Print(" RX ");     Serial.Print(packetSize);     Serial.println(" Bytes");     Serial.Print("Device ID");
#endif      first read the device id        for (uint8_t Ⅰ=0; Ⅰ<6;Ⅰ++){       Daniel[Ⅰ]=Lora.Read();
#ifdef Debug       Serial.Printf("-%02x",Daniel[Ⅰ]);
#endif     }
#ifdef Debug     Serial.println();
#endif     Calculate residual package     packetSize -= 6;     check if the device is registered     devnr = findDevice(Daniel);     If (devnr >= 0)  {       if yes, we set the time stamp for the last message and       read the data       Devices[devnr].Last = getLocalTime();       writeConfiguration("/configuration.csv");       while (packetSize > 0) {         Channel number = device number * 16 + device channel         Channel = Lora.Read() + devnr*16;
#ifdef Debug         Serial.Printf("Channel: %02x",Channel);
#endif         type of channel         Type = Lora.Read();
#ifdef Debug         Serial.Printf("Type: %02x",Type);
#endif         determine the length of the data packet and whether the channel is an actuator         Output = False;         Switch(Type) {           Case LPP_DIGITAL_INPUT : Len = LPP_DIGITAL_INPUT_SIZE - 2; Break;           Case LPP_DIGITAL_OUTPUT : Len = LPP_DIGITAL_OUTPUT_SIZE - 2; Output = True; Break;           Case LPP_ANALOG_INPUT : Len = LPP_ANALOG_INPUT_SIZE - 2; Break;           Case LPP_ANALOG_OUTPUT : Len = LPP_ANALOG_OUTPUT_SIZE - 2; Output = True; Break;           Case LPP_LUMINOSITY : Len = LPP_LUMINOSITY_SIZE - 2; Break;           Case LPP_PRESENCE : Len = LPP_PRESENCE_SIZE - 2; Break;           Case LPP_TEMPERATURE : Len = LPP_TEMPERATURE_SIZE - 2; Break;           Case LPP_RELATIVE_HUMIDITY : Len = LPP_RELATIVE_HUMIDITY_SIZE - 2; Break;           Case LPP_ACCELEROMETER : Len = LPP_ACCELEROMETER_SIZE - 2; Break;           Case LPP_BAROMETRIC_PRESSURE : Len = LPP_BAROMETRIC_PRESSURE_SIZE - 2; Break;           Case LPP_GYROMETER : Len = LPP_GYROMETER_SIZE - 2; Break;           Case LPP_GPS : Len = LPP_GPS_SIZE - 2; Break;           Default: Len =  0;         }         if the channel is not an actuator, we reset the message buffer to 1         so that the data is sent to the MQTT server at the next opportunity         If (!Output) messages[Channel].New =1;         messages[Channel].Type = Type;         Remaining package = 2 less because channel and type were read         packetSize -= 2;
#ifdef Debug         Serial.Print("Data:");
#endif         now we read the received data with the determined length         for (uint8_t Ⅰ=0; Ⅰ<Len; Ⅰ++) {           Dat = Lora.Read();           for actuators, we don't remember any data           If (! Output) messages[Channel].Data[Ⅰ] = Dat;
#ifdef Debug           Serial.Printf("-%02x",Dat);
#endif           Reduce the remaining package by one           packetSize --;         }
#ifdef Debug         Serial.println();
#endif       }       Update status       loraCnt++;       loraLast = getLocalTime();       Display();     } else {       The device is not registered        we remember the device id to display it for registration       for (uint8_t Ⅰ = 0; Ⅰ<6; Ⅰ++) Unknown[Ⅰ] = Daniel[Ⅰ];       newGeraet = True;       newGeraetType = 0; LoRa Device     }     Part two Send response to LoRa device     Delay(100);     Lora.beginPacket();     at the beginning, the device id     Lora.Write(Daniel,6);     we check if we have output data for the current LoRa device     Int devbase = devnr*16;     for (Int Ⅰ = devbase; Ⅰ<devbase+8; Ⅰ++) {       depending on the type of digital or analog data       Switch (messages[Ⅰ].Type) {           Case LPP_DIGITAL_OUTPUT : Lora.Write(Ⅰ-devbase);             Lora.Write(messages[Ⅰ].Type);             Lora.Write(messages[Ⅰ].Data,1);
#ifdef Debug             Serial.println("Digital Output");
#endif             Break;           Case LPP_ANALOG_OUTPUT :  Lora.Write(Ⅰ-devbase);             Lora.Write(messages[Ⅰ].Type);             Lora.Write(messages[Ⅰ].Data,2);
#ifdef Debug             Serial.println("Analog Output");
#endif             Break;       }     }          Int lstatus = Lora.endPacket();
#ifdef Debug     Serial.Print("Send Status = ");     Serial.println(lstatus);
#endif   }
}

Function to read the configuration
Void readConfiguration(Const Char *Fn) {   uint8_t Ⅰ = 0;   String Tmp;   Char Hex[3];   If (!SPIFFS.Exists(Fn)) {     does not yet exist then generate     writeConfiguration(Fn);     Return;   }   File Q = SPIFFS.Open(Fn, "r");   If (!Q) {     Serial.println(Q("ERROR:: SPIFFS Can't open configuration"));     Return;   }   while (Q.available() && (Ⅰ<MAXDEVICE)) {     Tmp = Q.readStringUntil(',');     Devices[Ⅰ].Active = (Tmp == "1");     Tmp = Q.readStringUntil(',');     Devices[Ⅰ].Service = Tmp.toInt();     Tmp = Q.readStringUntil(',');     for (uint8_t J=0; J<6; J++){       Hex[0]=Tmp[J*3];       Hex[1]=Tmp[J*3+1];       Hex[2]=0;       Devices[Ⅰ].Id[J]= (Byte) strtol(Hex,Null,16);     }     Tmp = Q.readStringUntil(',');     Devices[Ⅰ].Name = Tmp;     Tmp = Q.readStringUntil(',');     Devices[Ⅰ].Last = Tmp;     Ⅰ++;   }    }

Void Setup() {   Initialize device storage   for (uint8_t Ⅰ =0; Ⅰ<MAXDEVICE; Ⅰ++) Devices[Ⅰ].Active = 0;   OLED Display Initialize   pinMode(16,Output);   digitalWrite(16, Low);   Delay(50);    digitalWrite(16, High);   Display.Init();   Display.flipScreenVertically();   Display.setFont(ArialMT_Plain_10);   Display.setTextAlignment(TEXT_ALIGN_LEFT);   Start serial interface   Serial.Begin(115200);   while (!Serial);    Serial.println("Start");   Flash File syastem   If (SPIFFS.Begin(FORMAT_SPIFFS_IF_FAILED)) Serial.println(Q("SPIFFS Loaded"));   Read in the configuration   readConfiguration("/configuration.csv");     initMessageBuffer();   Initialize SPI and LoRa   Spi.Begin(5,19,27,18);   Lora.setPins(Ss,Rst,DI0);   Serial.println("LoRa TRX");   If (!Lora.Begin(Band)) {     Serial.println("Starting LoRa failed!");     while (1);   }   Lora.enableCrc();   Serial.println("LoRa Initial OK!");   Delay(2000);   Connect to the Wi-Fi and MQTT Server   Serial.println("Connect Wi-Fi");   Cayenne.Begin(MQTT_USER, MQTT_PASSWORD, MQTT_CLIENTID, WIFI_SSID, WIFI_PASSWORD);   Serial.Print("IP address: ");   Serial.println(Wifi.localIP());   Initialize Web Server   Server.On("/", handleRoot);   Server.Begin();   Synchronize clock with time server   configTime(GMT_OFFSET_SEC, DAYLIGHT_OFFSET_SEC, NTP_SERVER);   Output current time   Serial.println(getLocalTime());


}


Void Loop() {   Display();   Check LoRa Interface for data   readLoRa();   communicate with Cayenne MQTT Server   Cayenne.Loop(1);   Serving Web Server   Server.handleClient();

}

Send data from the message buffer to the MQTT server
CAYENNE_OUT_DEFAULT()
{   Boolean Output = False;   Boolean sentData = False;
#ifdef Debug   Serial.println(getLocalTime());   Serial.println("Cayenne send");
#endif   for (Int Ⅰ = 0; Ⅰ<MAXCHANNELS; Ⅰ++) {     send only new messages     If (messages[Ⅰ].New == 1) {
#ifdef Debug       Serial.Printf("Send MQTT Type %i'n",messages[Ⅰ].Type);
#endif       send data depending on type       Switch (messages[Ⅰ].Type) {           Case LPP_DIGITAL_INPUT : Cayenne.digitalSensorWrite(Ⅰ,messages[Ⅰ].Data[0]); Break;           Case LPP_DIGITAL_OUTPUT : Output = True; Break;           case LPP_ANALOG_INPUT : Cayenne.virtualWrite(i,(messages[i].daten[0]*256 + messages[i].data[1])/100,"analog_sensor",UNIT_UNDEFINED); break; break;           Case LPP_ANALOG_OUTPUT : Output = True; Break;           Case LPP_LUMINOSITY : Cayenne.luxWrite(Ⅰ,messages[Ⅰ].Data[0]*256 + messages[Ⅰ].Data[1]); Break;           Case LPP_PRESENCE : Cayenne.digitalSensorWrite(Ⅰ,messages[Ⅰ].Data[0]); Break;           Case LPP_TEMPERATURE : Cayenne.celsiusWrite(Ⅰ,(messages[Ⅰ].Data[0]*256 + messages[Ⅰ].Data[1])/10); Break;           Case LPP_RELATIVE_HUMIDITY : Cayenne.virtualWrite(Ⅰ,messages[Ⅰ].Data[0]/2,TYPE_RELATIVE_HUMIDITY,UNIT_PERCENT); Break;           Case LPP_ACCELEROMETER : Cayenne.virtualWrite(Ⅰ,(messages[Ⅰ].Data[0]*256 + messages[Ⅰ].Data[1])/1000,"gx","g"); Break;           Case LPP_BAROMETRIC_PRESSURE : Cayenne.hectoPascalWrite(Ⅰ,(messages[Ⅰ].Data[0]*256 + messages[Ⅰ].Data[1])/10); Break;           case LPP_GYROMETER : len = LPP_GYROMETER_SIZE - 2; break;           case LPP_GPS : len = LPP_GPS_SIZE - 2; break;       }       If (!Output) {         messages[Ⅰ].New = 0;         sentData = True;       }            }   }   If (sentData) {     Update status     cayCnt++;     cayLast = getLocalTime();     Display();   }

}

CAYENNE_IN_DEFAULT()
{   uint8_t * Pdata;   Int Val;   Int Ch = request.Channel;
#ifdef Debug   Serial.println("Cayenne recive");   Serial.Printf("MQTT Data for Channel %i = %s"n",Ch,Getvalue.asString());
#endif   Switch (messages[Ch].Type) {       Case LPP_DIGITAL_OUTPUT : messages[Ch].Data[0] = Getvalue.asInt();         messages[Ch].New = 1;         Break;       Case LPP_ANALOG_OUTPUT :  Val = round(Getvalue.asDouble()*100);         messages[Ch].Data[0] = Val / 256;         messages[Ch].Data[1] = Val % 256;         messages[Ch].New = 1;         Break;   }    }