Frostwächter mit ESP8266 und MicroPython - Teil 3

This article is also available as a PDF document on German and Translations.
This episode is also available as pdf document in english and english.

Today we are going to build an Android app for controlling the Freeze Guardian at close range around the WiFi router. There is a web server for remote access, which can be requested by (almost) any browser. "almost" because Firefox makes various noises when the request is made to a server that does not work with the HTTPS layer, such as Ubuntu16.04 LTS on my 32-bit machine. As of Ubuntu 18, only 64-bit systems are supported. But, as important as our data is now again, not that you would have to transmit them encrypted. So welcome to the 3rd part of the series about the

Freeze Guardian - Guardian app and WWW access

Figure 1: Android app for control

Figure 1: Android app for control

To get a basic overview of the desired network structure, I start with a graphic. In previous The Linux computer had already contacted the ESP8266 via UDP. Even the Windows PC could already communicate with the ESP8266 via the program Packetsender.exe, also via UDP. Messages from the ESP8266 were sent to both PCs. The ESP8266 runs in station mode and is therefore not directly accessible as an access point. So all the traffic runs through the WLAN access point - as far as the radio is concerned. In the graph, these are the light green arrows. Another one Part of the conversations happens mainly or exclusively via cables - the arrows in medium green. The traffic between the smartphone and the web server on Linux is dark green and also mixed. The small, pale green arrow stands for the interprocess communication between the UDP client and the TCP web server on the Linux box. And it is precisely with the latter that we begin today.

Figure 2: Network structure

Figure 2: Network structure

The web server on the Linux box

While the UDP client on Linux performs scheduled and asynchronous work in the background, the web server uses TCP to ensure that data from the ESP8266 is also available outside the local network. UDP is a fast, lightweight protocol that works without a connection. As a result, the data traffic is not secured and this in turn means that data packets can be lost or falsified.

If we want to access data from our ESP8266 from outside the LAN- WLAN area, this must be done via TCP and a fail-safe connection.

The web server on the Linux box provides the background for this. The server can be reached from (almost) any browser, in the WLAN area directly via the in-house, private IP address and worldwide with the help of a dynamic DNS service such as:

  • YDNS. ...
  • FreeDNS. ...
  • Securepoint DynDNS. ...
  • Dynu. ...
  • DynDNS service. ...
  • DuckDNS. ...
  • No IP.

A plain text URL can be obtained via these services (free of charge). The new IP address assigned by the provider every day is automatically connected to the URL by the service. Our web server is therefore always accessible at the same URL.

I have already explained the "fast" at the beginning. As with the UDP client on the Linux machine, the connection to the WLAN access point is no longer necessary with the web server, as there is a cable connection. The socket ("10.0.1.111",9002) is set up, with one exception, in the same way as with UDP. The only difference is the bold line. TCP works with streams, UDP with datagrams.

IPS="10.0.1.111"
portNumS=9002
print("Request server socket")
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
server.settimeout(1)
server.bind(('', portNumS))    # bind to local IP and port number 9192
server.list(5)              # Accept up to 5 incoming requests
print("Receive requests on {}:{}".format(IPS,portNumS))

# **************** Socket Setup Client ********************
IPC="10.0.1.111"
portNumC=9004  #
print("Request client socket")
client = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
client.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
client.settimeout(2)
client.bind(('', portNumC))    # bind to local IP and port number 9192
print("Send requests to {}:{}".format(IPC,portNumC))
# ****************** Target: Klimaagent client9191 ****************
targetPort=9001
target=("10.0.1.111",targetPort) # UDP client on Linux111

For internal computer communication, we set up the UDP socket ("10.0.1.100",9004). Both sockets are set up non-blocking, this is ensured by the timeout. Speaking names are very useful for the three addresses, so the third address also has one. It addresses the UDP client in the Linux PC as the target. This is where the web server sends its internal requests / commands and from there it receives its information, which it passes on to the browser.

When the sockets are ready, it goes into the endless loop of the server. server.accept() check briefly whether a connection request has been received. If there is nothing, an exception is thrown, which is try is intercepted. The corresponding except at the very end, without an executable command with passport undergo. The only point here is that the exception does not cancel the program.

If there is a request from a browser, then a communication socket c of accept and the address addr the requesting client. Over the socket c will the rest of the communication be handled, server is thus free again for further inquiries. Five of them can be served at the same time, so we determined it above.

while 1:                      # Infinite loop
   try:
     c, addr = server.accept()  # Accept web request
     print('Got a connection from {}:{}\n'.\
            format(addr[0],addr[1]))
     rawRequest=c.recv mode(1024)  # (A)
     # rawrequest is a bytes object
     # and must be decoded as a string
     #, so that string methods
     # can be applied to it.
     try:
       request = rawRequest.decode("utf-8")  # (B)
       getPos=request.find("GET /")
       if request.find("favicon")==-1:  #(C)
           print("**********************************")
           print("Position:",getPos)
           print("Request:")
           print(request)
           print("**********************************")  #(D)
           pos=request.find(" HTTP")
           if getPos == 0 and pos != -1:  # (Email)
               query=request[5:pos]  # (F)
               print("*****QUERY:{}******\ n\n".format(query))
               response = web_page(query)  # (Gram)
               print("---------\next",response,"\next-----------")
               c.send('HTTP/1.1 200 OK\n'.encode())  #(Houses)
               c.send('Content-Type: text/html\n'.encode())
               c.send('Connection: close\n\n'.encode())
               c.sendall(response.encode())
           else:   # (Hunter)
               print("############\ nNOT HTTP\n############")
               c.send('HTTP/1.1 400 bad request\n'.encode())
       else:
           print("favicon request found")
           c.send('HTTP/1.1 200 OK\n'.encode())
     except:
       request = rawRequest
       c.send('HTTP/1.1 200 OK\n'.encode())
     c.close()
   except:
       passport

(A)

We get from c the request, a byte object that contains more overhead than payload, and convert it to a string (B), which is easier to act on.

(C)

The annoying habit of browsers, a file favicon.requesting ico leads to problems and is beaten off by the if construction.

too (D)

These print commands and all the following only help to track and control the functioning of the server in the terminal window. They can be removed without problems if everything goes perfectly.

too (E)

If there is a "GET" at the very beginning of the request and an "HTTP" a little further back, then it may be one that the server should answer. Requests of this type have one of the following forms.

GET / HTTP/1.1
Host: 10.0.1.100:9002
Connection: keep-alive

GET /?N HTTP/1.1
Host: 10.0.1.100:9002
Connection: keep-alive
………

or
GET /?B;heizenAn HTTP/1.1
Host: 10.0.1.100:9002
Connection: keep-alive
………

The parts shown in bold contain the commands to the server, which we filter out (F).

too (G)

As always, the decoding is performed by the parser function, which is also responsible for creating the webpage at the same time and is therefore also called, web_page(). We hand over to her the parsed command query. response receives the returned web page text.

too (H)

This is followed by the output of the HTML header and page code.

too (J)

If the request did not meet our expectations, 400 "bad request" will be sent back to the browser.

Let's go to the Parser more.

can)

The time module offers slightly different methods in CPython compared to MicroPython. For example, the strftime() method provides the data on the date and time in conjunction with a freely definable string structure. Here, for example, "Time: 06.11.2021 09:14:01\n".

too (L)

We request the last saved temperatures from the UDP client on Linux and determine the length of the command string.

too (M)

If the instruction length is 0 or 1, then the else branch returns only the temperatures.

10.0.1.100   or  (length=0)
10.0.1.100/    or (length=0)
10.0.1.100/?   (length=1)

However, if 2 or more characters are included, then the question mark should be followed by either an "N" or a "B;COMMAND_AN_DEN_UDP_CLIENT". In addition to the temperatures, an "N" also requests the last 5 lines from the messages file.

too (P)

A B must be followed by a delimiter (;) and then one of the following commands:

  • getTemp
  • sendTemp
  • heizenAn
  • heizenAus
  • autoOn
  • AutoOff
  • exit
  • reboot

The Linux UDP client forwards these commands to the ESP8266 UDP server and then waits for the response to be built into the HTML page.

def web_page(command):
   date=strftime("

Time: %d.%m.%Y %H:%M:%S

\n"
) # (Can)

   tempString=getTemperatures()  #(Countries)
   evaluate=""
   length=len(command)
   print("Length:",length)
   if length>1:   # (M)
       command=command[1:]
       print("Command:{},\
Length:{}".format(command,len(command)))
       # Execute command
       # Get an answer
       cmdReply="
\n"

       if command[0]=="Next":   # (Next)
           tempString=getMessages()
       evaluate=tempString+cmdReply
       if command[0]=="B":   # (P)
           evaluate=doCommand(command[2:])
           evaluate=tempString+evaluate+cmdReply
   else:
       print("Temperatures environment{}".format(tempString))
       evaluate=tempString
       # temperatures only
   html1 = """ # (Q)

initial scale=1">


Climate garden house

"""

   html1=html1+ date + evaluate

   html2 = "# # # # # # # # # # # # # # # # # # # # # # #"
   html3 ="""













Automatic On


Automatic Off


Heating On


Heating Off


Query status

"""

   html9 = " "
   HTML = html1 + html2+html3+html9
   #print("Answer: \n",html)
   return HTML

too (Q)

From here, the HTML page is built, finally assembled and returned.

A number of other functions perform more complex tasks.

getTemperatures() requests the last record from the data-gh file from the Linux UDP client and builds an HTML snippet from it, which is returned.

def getTemperatures():
   # Get temperatures
   print("Request temperatures")
   client.sendto("S".encode(),target)
   try:
       sleep(0.2)
       answer,cor=client.recvfrom(256)
       answer=answer.decode().strip("\next")
       date,envir,house,box=answer.split(";")
       temps="

Last update:{}

Environment: {}
\nGartenhaus: {}
\NbOx: {}

</B>\n"\
.format(date,envir,house,box)
   except:
       temps="NO DATA
"

   return  temps # evironment,house,box

Proceed similarly getMessages() with the requested data from the file messages.

The message is cleaned of flanking newline characters (\n) and the temperature string at "_" is separated from the messages. The temperature string is separated by the ";" and the date, time and temperature values are formatted in HTML.

The news in mesg we split up the end-of-line characters "\n" and generate a list from them mesgs, it is the basis for the HTML string that we build from it in the for loop.

def getMessages():
   # Get temperatures and news
   print("Request temperatures and news")
   client.sendto("Next".encode(),target)
   try:
       sleep(0.2)
       answer,cor=client.recvfrom(512)
       answer=answer.decode().strip("\next")
       temp,mesg=answer.split("_")
       date,envir,house,box=temp.split(";")
       temps="

Last update:{}

Environment: {}
\nGartenhaus: {}
\NbOx: {}
\n"
\

          .format(date,envir,house,box)
       mesgs=mesg.split("\next")
       mesg="

Latest news:

"

       for m in mesgs:
           mesg=mesg+m+""
           print(mesg)
       mesg=mesg+"
"
       all = temps+mesg
   except:
       all="DATA INCOMPLETE
"

   return  all # evironment,house,box

function doCommand() also requests the temperatures and also gives the command in the parameter cmd forward to the Linux UDP client. We wait for its response and return it as an HTML sequence.

def doCommand(cmd):
   # Get temperatures
   OnOff=["from","to"]
   print("Request temperatures")
   client.sendto(("B;"+cmd).encode(),target)
   try:
       sleep(1.0)
       answer,cor=client.recvfrom(256)
       answer=answer.decode().strip("\next")
       result=answer[2:]
       print("***************",answer,"***************")
       if answer[0]=="South":
           result="Heater "+OnOff[int(result[0])]+\
                  " Automatic "+OnOff[int(result[1])]
       temp="

Command: {}

{}
\n"
.\

            format(cmd,result)
   except:
       temp="NO DATA TIMEOUT
"

   return  temp # evironment,house,box

To send and receive using the sendto() and recvfrom() commands, we must note that in CPython bytes objects are sent and received, not strings. therefore, when receiving, the bytes object must be decoded to the string, and the string must be encoded before sending. In MicroPython, this happens implicitly when sending.

The HTML code sent to the browser contains, in addition to the temperature data, a table with the links that send command sequences to the web server. This saves typing in the address bar of the browser by hand and thus prevents potential typos.

<HTML>
   <head>
   <meta name="viewport" content="width=device-width,
      initial-scale=1">
   </head>
   <body>
   <h1>Climate Garden house</h1><H2>
       Time: 06.11.2021 20:22:53</H2>
   <H3>Latest Stand:06.11.2021 20:20</H3>
   <B>Environment:  18,50<BRUNETTE>
      Garden house:  18,75<BRUNETTE>
      Box:  18,44<BRUNETTE></B>
   # # # # # # # # # # # # # # # # # # # # # # #
   <table border=2 cellspacing=2>
   <dream>
   <td><a href='http://10.0.1.100:9002 /?B;autoOn'>
       <H3>Automatics To</H3> </a></td>
   <td><a href='http://10.0.1.100:9002 /?B;AutoOff'>
       <H3>Automatics From</H3> </a></td>
   </dream>
   <dream>
   <td><a href='http://10.0.1.100:9002 /?B;Heating>
       <H3>Heater To</H3></a></td>
   <td><a href='http://10.0.1.100:9002 /?B;Heating OFF'>
       <H3>Heater From</H3> </a></td>
   </dream>
   <dream>
   <td><a href='http://10.0.1.100:9002 /?B;status'>
       <H3>Status query</H3> </a></td>
   </dream>
   </table></body> </HTML>

Figure 3: Website

Figure 3: Website

You can download the text of the Linux programs here:

client9001.py

converttemp.py

archivate.py

webserver.py

The previous articles on the subject of frostwatchers can be found here:

Part 1 - Hardware and Programming of the ESP8266 in MicroPython

Part 2 - UDP Client on the Linux Machine (Ubuntu 16.04 LTS)

The advantage of the web server is clearly the worldwide accessibility, provided DDNS. By the way, the server can also be trained to send the entire daily file or archived files as web pages. To do this, we would only have to read the web server directly from the respective file and have each line sent as a line of text in an HTML framework. It's not difficult, try it! No more know-how is necessary than already in the two files client9001.py and webserver.py stick.

The Android app

But now we turn to the Android app, it can do something that the website can't. It speaks directly to the ESP8266 and reacts to alarm signals from it. As a result, an announcement text and a siren let us know that the plants are to be watered. Let's see what is needed for the app, in addition to the Android phone of course.

For the mobile phone

AI2-Companion from the Google Play Store.

For the mobile app

http://ai2.appinventor.mit.edu

https://ullisroboterseite.de/android-AI2-UDP/UrsAI2UDP.zip

Install and use App Inventor - Detailed instructions

The finished development (freezeguardian.aia)

The finished app (freezeguardian.apk)

We create the app with the help of the tool AppInvertor2, which can be used under the MIT license as free software without installation via a browser (e.g. Firefox). This means that the application does not have to be installed on the PC if a WLAN connection is available. How to deal with it, I have described in great detail here, that's why I'm not going into it in more detail now. Also the use of the UDP extension for this tool from Ulli's robot page will be explained in detail there.

Programming with the Appinventor is event-driven and object-oriented. This means that we react with our program to individual events such as clicks on buttons, the expiration of a timer, the receipt of a UDP message and so on. All this happens in a modular system.

Let's start the Appinventor in Firefox or Chrome. If you are using the software for the first time, I strongly recommend working through it my introduction to this topic.

http://ai2.appinventor.mit.edu

Figure 4 shows the finished layout that we want to create right away:

Figure 4: The Viewer window

Figure 4: The Viewer window

From the column Range, to the left of the Viewer let's get the objects we need by drag and drop. "Forest Guard" is a Record label, "Manual heating control" also. To the right of the viewer we have the windows Components and Properties. Components displays the hierarchy of the arranged elements. In Properties the properties are set. "Frostwächter" has the font size 20 and the color light blue, is visible and centered. Also, the "horizontal orientation" property of the "screen1" it's centered.

Figure 5: layout folder

Figure 5: layout folder

The structure of the blocks Heater and Automatics is identical. We start with a "VerticalArrangement". We put it in there."Horizontal arrangement".

From the folder "User Interface" let's get two Labels and put them over the horizontal arrangement. Into this come two Buttons.

Figure 6: User interface

Figure 6: User interface

Using the properties, we set the width of the elements, the font size, font color and the background to our liking. For example, the width is up "Automatic", then the width of the field adapts to the width of the text. "Heating" is set to 60%, "OFF" to 50 and the buttons to 24. The height is "Automatic". All alignments are "center". In addition to the solid colors, you can get by "Custom" choose RGB colors and transparency from a color area.

Figure 7: Colors

Figure 7: Colors

The "sceren1" shall "Scrollable" switched so that you can roll into the lower area.

For the temperatures, a "VerticalArrangement" a horizontal with 6 Lables and among them a Button "UPDATE". "GET STATUS" has a button and below it two labels.

Figure 8: Lower screen area

Figure 8: Lower screen area

UDP messages are displayed in the following label, feedback from the ESP8266 in the green label field and error messages in the red. The gray button below shows messages from the ESP8266 that reach the app without prior request to the controller. These are messages from the humidity sensor. Whenever the condition tilts and as a reminder at the end of each day, we are informed of the moisture state of the planting soil. In the case of "too dry" messages, a voice prompt informs about this and a siren wakes us up from our dreams of a contented plant paradise. The alarm can be switched off via the button. "RESTART LISTENER" informs us about the state of the UDP receiver.

Do not be alarmed if an error message with the number 2 or 6 appears. This happens from time to time during development when changing properties and/or adding or removing elements. The mobile phone then tries to restart the app, which sometimes happens without the UDP socket having been closed beforehand. The start with the same IP and port number then goes wrong, and the connection between the mobile phone and Appinventor must be cut and re-established.

In addition to the previously listed, a whole series of "invisible" components can still be recognized at the bottom of the image in the "Viewer".

Figure 9: Invisible elements

Figure 9: Invisible elements

From left to right, these are the UDP receiver and transmitter, TextToSpeech for voice output, two timers, a sound playback called siren and another timer for alarm repetition.

Once all components have been arranged, we switch from the "Designer" to the "Blocks", left-click on the far right in the action bar

Figure 10: Menus

Figure 10: Menus

Each event is now provided with an event handler by an arrangement of corresponding bricks. There are various short handlers, a few medium-gorgeous and an absolute monster handler. The latter corresponds to the parser routines in our Python programs. In general, we encounter the same structures here several times as in the MicroPython programs, the indentations through the brackets, the reaction to events, etc.

We start with the definition and declaration of the required global variables. In addition, we create the procedure for obtaining the temperature values. The definition blocks of the variables are retrieved from the folder Variable pulled out the occupancy from Text, also join comes from there.

We take the procedure bracket from the folder Procedures. Diving into blocks at the very bottom UDPXmitter1 up, from there we pull a call-Block the viewer. The text constant getTemp send to the ESP8266. Because of course the answer will not be available immediately, we set the alarm clock clkProgressDelay at 1000ms and make him sharp. We'll come back to that later.

Figure 11: Blocks Variables and Functions

Figure 11: Blocks Variables and Functions

Every program needs an initialization phase, here is ours. With the screen setup, things also happen in the when screen1 initialize-Bracket.

Figure 12:Block Initialization

Figure 12:Block Initialization

The UDP receiver is stopped, the transmitter is aimed at the ESP8266 as the target, and the receiver is set to the port number 9001. Then we query the status of the recipient and use join to create a message for the label lblUDPmessage. Admittedly, this looks monstrous, and in MicroPython this goes shorter. But this results in a product that runs on Android, without us having to have any idea about the programming under this operating system.

We continue with the reactions to buttons and timers. Six of the brackets have the same or at least a similar structure, the treatment of the buttons. As with the others, the when btnAutomatikOn.Click the content of the error display lblError deleted and the corresponding command string sent to the ESP8266. The order to update the temperatures is more complex and is determined by the procedure requireTemperatures initiated.

Figure 13: Block Events

Figure 13: Block Events

The situation is similar with the timers. In any case, the timer must be defused after the response. Shall clkProgressDelay expired, the handler deletes the system message in lblSystemMessages and gives the command to send the temperature values to the ESP8266. Error messages are deleted when the timer clkErrorDelay expires.

The RESTART LISTENER button triggers almost the same actions as during initialization.

Figure 14: Blocks Alarm handling

Figure 14: Blocks Alarm handling

These three blocks handle the alarm case. The alarm triggering is the responsibility of the alarm procedure. The alarm timer is initially set to 3 seconds and armed. The voice message "The plants need watering" is output via the mobile phone speaker. In order for language to really come out, we set the properties Country and Language the right values are already set in the designer. If they do not fit, only silence comes out of the speaker.

Figure 15: TextToSpeech

Figure 15: TextToSpeech

When the alarm timer expires, the sound of the siren starts. To do this, a sound file in MP3 format must be uploaded to the mobile phone in the designer. Free sound files can be found on the Internet a lot (for example at Salamisound).

Figure 16: Siren

Figure 16: Siren

Only when the alarm is active, the button is also armed. With this we turn off the annoying noise. The sound is stopped, the timer is disabled, a reminder is issued for watering the plants and the button is marked with the warning message "TOO DRY".

The parser

Figure 17: Blocks Start Parsing

Figure 17: Blocks Start Parsing

Parsing is started when a message arrives in the buffer of the UDP receiver. when UDPListener.dataReceived then responds, the payload catches up response and queries the sending address. This will be in the label lblUDPmessage represented. The received text is output in the label for system messages. Finally, we find out the type of the message by isolating the first letter.

Several if-elseif blocks follow and filter out the individual responses of the ESP8266, as we have already done in the MicroPython programs. The type T is the most interesting because it shows the object list in the application.

Figure 18: Block Temperatures

Figure 18: Block Temperatures

block split text divides the content of the response from the ESP8266 into individual strings at the dashes and assigns them to the global variables temperatures environment as a list too. This corresponds to the MicroPython command

temperatures environment = response.split(";")

We now assign the list elements individually to the output fields. From the Lists-Let's get folders for this purpose select list item. At cunning let's link our list temperatures environment a and b index the index that starts with 2, because 1 is the type. At the box temperature, the "\n" must be cut off at the end. If the temperature in the box is less than 4 °C, the background color of the screen is set to blue, at higher values - to green.

The blocks for heating and automatic are almost identical and not very demanding. They are executed when guy an "H" is detected. The only new element is contains text from the folder Text. It examines the string in text the occurrence of the substring in piece.

Figure 19: Block heating

Figure 19: Block heating

The rest of the parser also no longer brings moving new things. I present the blocks only for the sake of completeness.

The status query:

Figure 20: Block Status

Figure 20: Block Status

In the rest, when changing from wet to dry, the alarm procedure is called, which we have already discussed.

Figure 21: Block Moisture

Figure 21: Block Moisture

Now it's up to you whether you want to place the objects in the designer yourself and arrange the blocks in the viewer yourself, or you can't wait to see the finished solution in function. One thing you have to keep in mind, however, to make it work, you have to adapt the socket data, IP address and port number, to your network.

Otherwise, I wish you a lot of fun programming other servers based on UDP or TCP or in connection with Appinventor 2. You received the tools for this by working through this project.

The previous articles on the subject of frostwatchers can be found here:

Part 1 - Hardware and Programming of the ESP8266 in MicroPython

Part 2 - UDP Client on the Linux Machine (Ubuntu 16.04 LTS)

Esp-8266Projects for advancedSensorsSmart home

Leave a comment

All comments are moderated before being published

Recommended blog posts

  1. Install ESP32 now from the board manager
  2. Lüftersteuerung Raspberry Pi
  3. Arduino IDE - Programmieren für Einsteiger - Teil 1
  4. ESP32 - das Multitalent
  5. OTA - Over the Air - ESP programming via WLAN