Make Faraday-APRS Persistent Across Network Failures #271

kb1lqc · 2017-09-10T20:02:53Z

BASE91 APRS Comment Telemetry

aprs.fi and a few other services support BASE91 telemetry transmission. This allows up to 13 bits of ADC telemetry per channel. This PR pulled in what was #269 so here is the information relevant to that PR. @reillyeon beat @el-iso to the punch on there. I appreciate the help from both!

Changelog

Imports aprslib and uses base91 functionality. Please note that this is note a standard BASE91 implementation and is a custom APRS version of base91. Qeue XKCD.
Removes use of sendTelemetry function as it is no longer needed
Only sends scaling values on first transmission to reduce unnecessary packet transmissions. This should be revisited later but seems good for now.
Sends a telemetrySequence number that is unbounded to sendPosition as BASE91 comment telemetry can support up to 16 bits of counting and this should be fine for the forceable future :)
sendPosition implements BASE91 Comment Telemetry with ADC0, ADC1, ADC3, ADC6, BOARDTEMP, and GPIO bits per Implement APRS BASE91 Comment Telemetry #266.

Raw packets which end in BASE91 comment telemetry

12 bit ADC data looks very nice!

Persistent APRS-IS Connection Across Network

This PR is based on #269 as it was branched from that PR repository and aims to improve reliability when connecting to APRS-IS servers across a network. As we've seen, often a few minutes to a few hours after starting APRS logging a network event will cause socket ERRNO 10054 or similar and faraday-aprs could not recover. This was annoying at best and at worst rendered the program not terribly useful.

Mind helping me out @reillyeon, if no time I can see if @kb1lqd has some free time in the near future :)

Changelog

Created a sendAPRSPacket function to consolidate sending of data to the socket and exception catching
Reconnects after a IOERROR in 2 second loop attempts

Issue Tickets

This PR addresses issues related to the following tickets Faraday-Software #119, Faraday-Software #190, Faraday-Software #199, Faraday-Software #201, Faraday-Software #262. A few of these tickets can likely be completely closed as a result of this testing and eventual PR!

Using aprslib with a simple import of base91 function to help create position comments that will send highly accurate data (up to 16 bits). Therefor this will help with using all 12 bits of the ADC on Faraday!.

all channels previously sent with telemetry packet are now encoded into BASE91 format. They are however not used yet.

Needed it for BASE91 APRS transmissions. Also commented out sendTelemetry.

I had to change up how the telemetry sequence counter worked. Also, we actually can count higher than 255 with BASE91 telemetry so we will do that. Also, this has a lot of development code but I will clean it up later as I let it run for a while.

He pointed out that I missed the width setting in his implementation of BASE91. It appears to be working! My bad...

Removed a bunch of dev code and added normal comments back in.

using integers encoded into BASE91 format I now also send IO data to APRS-IS with each position!

Decided to only send them once right now during initial loop through sending. This will hopefully reduce the number of times I have APRS.fi tell me I'm abusing the network :)

I changed them for development testing and never changed them back. This function is currently unused as it is.

Lots of development code in here but I can close down the aprs-is server (aprsc) and start it back up after indication of an error by faraday-aprs. The program just keeps on going after server starts back up.

Created a function that could be commong to all packet types. This prevents me from having to duplicate the same code a bunch in aprs.py

Removed old commented out code while also adding documentation for sendAPRSPacket.

While still commented out, the labels, parameters, and equations packets are now send on every 10th loop.

Went through all packets being sent and made them all use the sendAPRSPacket function. This helps clarify the design as well as provide fault detection and reconnection.

Fixed some errors in the docstring.

ADC names were wrong and I made them not wrong.

Running tests over the day I observed from a local aprsc server that network connectivity was lost for a few minutes at the same time that my APRS telemetry stopped. Investigating by disableing my own network connection for several faraday-aprs loops I observed the ERRNO 10054, ERRNO 11004, and ERRNO 9 errors. I found that while I closed the socket aprsSock in the connection code on an IOError I did not recreate a new socket for aprsSock which resulted in an indefinitely bad file descriptor (because it didn't exist) whenever I tried to reconnect. This should be fixed now as I added the automatic aprsSock creation in.

reillyeon · 2017-09-10T20:39:34Z

faraday/aprs.py

@@ -103,6 +104,11 @@ def aprs_worker(config, sock):
    # Local variable initialization
    telemSequence = 0

+    #dev
+    sock = ''


This line is unnecessary.

Removed, conn=False now used in the initialization just above to force it to initialize on first loop. Thanks for catching this!

reillyeon · 2017-09-10T20:40:58Z

faraday/aprs.py

+
+            # Just send labels, Parameters, and Equations every 10th loop
+            if telemSequence % 10 == 0:
+


nit: Extra blank line.

reillyeon · 2017-09-10T20:41:13Z

faraday/aprs.py

@@ -103,6 +104,11 @@ def aprs_worker(config, sock):
    # Local variable initialization
    telemSequence = 0

+    #dev
+    sock = ''


nit: This statement is unnecessary.

reillyeon · 2017-09-10T20:41:53Z

faraday/aprs.py

-                except IOError as e:
-                    logger.error(e)
+                status = sendAPRSPacket(socket, positionString)
+                return status


return sendAPRSPacket(socket, positionString)

Here and below.

All return status lines replaced with obviously better return sendAPRSPacket(). Thanks for observing this unnecessary addition of variables!

reillyeon · 2017-09-10T20:43:24Z

faraday/aprs.py

                logger.error(e)
+                aprssock.close()
+                aprssock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

            else:
                logger.info("Connection successful!")
                return aprssock
                break


Not this change, but break after return does nothing.

reillyeon · 2017-09-10T20:46:18Z

faraday/aprs.py

@@ -762,7 +786,8 @@ def main():
    """

    logger.info('Starting Faraday APRS-IS application')
-    sock = connectAPRSIS()
+    #sock = connectAPRSIS()
+    sock = ''


Why pass sock to aprs_worker at all if it calls connectAPRSIS() itself?

Good point. I used to need this but with the current implementation of the socket connections I do not. Removed!

Removed the #dev comment as well as manual setting of sock to a string. Instead I changed conn=False to force it to always initialize inside the loop.

Removing unnecessary use of status variable by directly returning output of sendAPRSPacket.

aprs_worker creates it's own socket connection so it's unnecessary to pass any reference to a socket file descriptor when starting up the thread!

reillyeon · 2017-09-10T22:03:25Z

faraday/aprs.py


-            sleep(10)  # Try to reconnect every 10 seconds
+            sleep(2)  # Try to reconnect every 10 seconds


Update the comment?

reillyeon · 2017-09-10T22:06:05Z

faraday/aprs.py

+
+        # add telemetry to comments
+        comment = comment + b91Tlm
+        altComment = altComment + b91Tlm


comment += b91Tlm altComment += b91Tlm

a += b is more efficient than a = a + b.

@reillyeon

Per @reillyeon a += b is more efficient than a = a + b. Therefore I changed it.

kb1lqc · 2017-09-10T23:05:58Z

@reillyeon if you're going to help pull in the code that is effectively PR #269 then I should bring over the changelong here too. I appreciate the help on these if you are doing both!

reillyeon · 2017-09-10T23:08:13Z

PR #269 plus the changes here look good to me.

kb1lqc · 2017-09-10T23:17:06Z

Learn something new everyday... By pulling in #271 github automatically merged #269 since it was based on the same branch. This is awesome and in retrospect obvious! This means that as we progress through a big task we can commit mini PR's on it and still have one big PR at the end. If one PR that is a combination of a few smaller PR's in merged then we no longer have to worry about those. I like that!

@kb1lqd for awareness.

kb1lqd · 2017-09-11T02:12:16Z

Whoa thanks all and now I learned something new too, very cool PR trait!

kb1lqc added 21 commits September 4, 2017 17:52

Initial addition of APRS BASE91 Code

dad1f90

Using aprslib with a simple import of base91 function to help create position comments that will send highly accurate data (up to 16 bits). Therefor this will help with using all 12 bits of the ADC on Faraday!.

ADC0, ADC1, ADC3, ADC6, and BOARDTEMP BASE91

62263c5

all channels previously sent with telemetry packet are now encoded into BASE91 format. They are however not used yet.

Added telemetry sequence to sendPosition

5ccd19a

Needed it for BASE91 APRS transmissions. Also commented out sendTelemetry.

Per @rossengeorgiev there is a width for BASE91

eb55358

He pointed out that I missed the width setting in his implementation of BASE91. It appears to be working! My bad...

Cleaned up APRS Position Telemetry Code

d132474

Removed a bunch of dev code and added normal comments back in.

Added IO to position comment data

0a57a3b

using integers encoded into BASE91 format I now also send IO data to APRS-IS with each position!

Only send Parameters, Labels, and Equations once

7ead5f3

Decided to only send them once right now during initial loop through sending. This will hopefully reduce the number of times I have APRS.fi tell me I'm abusing the network :)

Fixed up pytest PEP8 Failures

b82e2fd

Updated comments for sendPosition() function

161145c

Reversed 4095 hardcoded values in sendTelemetry

541f10c

I changed them for development testing and never changed them back. This function is currently unused as it is.

Just got reconnection working!

e68b408

Lots of development code in here but I can close down the aprs-is server (aprsc) and start it back up after indication of an error by faraday-aprs. The program just keeps on going after server starts back up.

Removed unnecessary comment in sendPositions

4db85fb

Simplified packet sending with sendAPRSPacket

8eafdf5

Created a function that could be commong to all packet types. This prevents me from having to duplicate the same code a bunch in aprs.py

Removed commented out code and added documentation

041a922

Removed old commented out code while also adding documentation for sendAPRSPacket.

Send all packets on 10th loop

bf2a6f1

While still commented out, the labels, parameters, and equations packets are now send on every 10th loop.

Implemented sendAPRSPacket() on all packets

0565340

Went through all packets being sent and made them all use the sendAPRSPacket function. This helps clarify the design as well as provide fault detection and reconnection.

Updated sendAPRSPacket documenation

982b169

Fixed some errors in the docstring.

Fixed pytest errors/warnings

ffe768d

Updated parameter names

5c47607

ADC names were wrong and I made them not wrong.

kb1lqc added bug enhancement labels Sep 10, 2017

kb1lqc added this to the Alpha Software milestone Sep 10, 2017

kb1lqc self-assigned this Sep 10, 2017

kb1lqc requested a review from reillyeon September 10, 2017 20:02

reillyeon requested changes Sep 10, 2017

View reviewed changes

kb1lqc added 3 commits September 10, 2017 14:25

Removed development code/comments

799b61f

Removed the #dev comment as well as manual setting of sock to a string. Instead I changed conn=False to force it to always initialize inside the loop.

Removed extra line under if statement

ac65bd6

Replaced return status with reture sendAPRSPacket

b354512

Removing unnecessary use of status variable by directly returning output of sendAPRSPacket.

kb1lqc added 3 commits September 10, 2017 14:33

Removed useless break statement after return

a0e6d73

Cleaned up commented code and sock initialization

ddcb2ff

Removed passing of sock variable to threads

75d6930

aprs_worker creates it's own socket connection so it's unnecessary to pass any reference to a socket file descriptor when starting up the thread!

reillyeon approved these changes Sep 10, 2017

View reviewed changes

Updated reconnect comment and BASE91 comment add

f529c01

Per @reillyeon a += b is more efficient than a = a + b. Therefore I changed it.

kb1lqc merged commit fd03b82 into FaradayRF:master Sep 10, 2017

kb1lqc mentioned this pull request Sep 11, 2017

Update APRS Application to Handle Network Errors Gracefully #119

Closed

kb1lqc deleted the issue119 branch September 11, 2017 06:10

This was referenced Sep 28, 2017

Long Duration APRS Failed - [Errno 32] Broken pipe #199

Closed

Proxy/Telemetry/APRS Error Long Duration Raspberry Pi 3 #190

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Faraday-APRS Persistent Across Network Failures #271

Make Faraday-APRS Persistent Across Network Failures #271

kb1lqc commented Sep 10, 2017 •

edited

Loading

reillyeon Sep 10, 2017

kb1lqc Sep 10, 2017

reillyeon Sep 10, 2017

kb1lqc Sep 10, 2017

reillyeon Sep 10, 2017

kb1lqc Sep 10, 2017

reillyeon Sep 10, 2017

kb1lqc Sep 10, 2017

reillyeon Sep 10, 2017

kb1lqc Sep 10, 2017

reillyeon Sep 10, 2017

kb1lqc Sep 10, 2017

reillyeon Sep 10, 2017

kb1lqc Sep 10, 2017

reillyeon Sep 10, 2017

kb1lqc Sep 10, 2017

kb1lqc commented Sep 10, 2017

reillyeon commented Sep 10, 2017

kb1lqc commented Sep 10, 2017

kb1lqd commented Sep 11, 2017


		# Just send labels, Parameters, and Equations every 10th loop
		if telemSequence % 10 == 0:


		sleep(10) # Try to reconnect every 10 seconds
		sleep(2) # Try to reconnect every 10 seconds

Make Faraday-APRS Persistent Across Network Failures #271

Make Faraday-APRS Persistent Across Network Failures #271

Conversation

kb1lqc commented Sep 10, 2017 • edited Loading

BASE91 APRS Comment Telemetry

Changelog

Persistent APRS-IS Connection Across Network

Changelog

Issue Tickets

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kb1lqc commented Sep 10, 2017

reillyeon commented Sep 10, 2017

kb1lqc commented Sep 10, 2017

kb1lqd commented Sep 11, 2017

kb1lqc commented Sep 10, 2017 •

edited

Loading