RTC vs NTP timekeeping?

[bvwelch]

A big attraction for me is no requirement for internet... So could we have some subset of timekeeping, given only a DS3231 ?
Admittedly it would be nice if NTP could set the initial value of the RTC.
ps timezones and NTP have been a challenge for me in the past.

[timmbogner]

Yep! My plan is to add RTC support after the NTP infrastructure is in place.

[svdrummer]

NTP has restrictions to stop flooding. Some people poll once per second. This get detected and sometimes an IP block gets put in place for a while. I guess that is fair.

[aviateur17]

If you were to plug in the time manually, with no Internet, would you want to do it via MQTT or via Serial access or some other way? If some other way what would you prefer? Web Interface would be nice but I'm not sure how hard that would be to code in. Would be nice to have a web Interface with serial debug output.

[svdrummer]

Just as a thought...I have a remote area system and i use a very cheap GPS serial output module. Even one of the fake Chinese one will work. i just get the time from the $GPRMC or $GNRMC sentences. This was you can poll every second without restrictions.
I just wrote a little function to convert to local time. Just a thought. The time will even work inside as you don't need full GPS lock for the time, just enough to get the almanac data ie, one sat and weak signal.

[aviateur17]

I've ordered an RTC and GPS module from AliExpress and they should be in here in a couple weeks. My plan is adding support for those after I receive the hardware. I'm hoping not to add any libraries for the GPS but will probably need to add Makuna/RTC for integration with external RTC's.

[aviateur17]

Just FYI - working on this now...

1707671932 | [2] GPS String: $GNRMC,171854.000,V,,,,,,,110224,,,N,V*23
1707671932 | [2] GPS Time: 171854 ($GNRMC)
1707671932 | [2] GPS Date: 110224 ($GNRMC)
1707671932 | [2] GPS String: $GNZDA,171854.000,11,02,2024,00,00*40
1707671932 | [2] GPS Time: 171854 ($GNZDA)
1707671932 | [2] GPS Date: 11022024 ($GNZDA)

[timmbogner]

Niceeee. How are you setting up the additional UART interface?

[cando-daves]

I've ordered an RTC and GPS module from AliExpress and they should be in here in a couple weeks. My plan is adding support for those after I receive the hardware. I'm hoping not to add any libraries for the GPS but will probably need to add Makuna/RTC for integration with external RTC's.

Can you provide links to the units you ordered please?

[aviateur17]

RTC module: RTC DS3231 AT24C32 Precision RTC

https://www.aliexpress.us/item/2251832642943377.html?spm=a2g0o.order_list.order_list_main.30.8a9d1802051tRa&gatewayAdapt=glo2usa

GPS module: (link below is no longer valid)
(reminder that's why it's a good idea to print the page soon after purchasing)
TZT GPS+BDS Beidou dual-mode module, flight control satellite positioning navigator, ATGM336H replacement, for NEO-M8N

AT6558 based
TZT GPS+BDS Beidou dual-mode module, flight control satellite positioning navigator, ATGM336H replacement, for NEO-M8N - AliExpress.pdf

ATGM336H-5N

5V
RS232

https://www.aliexpress.us/item/3256801326060593.html?spm=a2g0o.order_list.order_list_main.40.8a9d1802051tRa&gatewayAdapt=glo2usa

[aviateur17]

Niceeee. How are you setting up the additional UART interface?

fdrs_gateway_config.h

// GPS UART Interface
#define USE_GPS
#define GPS_RXD 25
#define GPS_TXD 27

fdrs_gateway.h

#ifdef USE_GPS
  begin_gps();
#endif

fdrs_gateway_serial.h

#if defined (ESP32)
#define UART_IF Serial1
#define GPS_IF Serial2
#else
#define UART_IF Serial
#endif

void begin_gps() {
  GPS_IF.begin(9600, SERIAL_8N1, GPS_RXD, GPS_TXD);
}

Serial Output:

10 | [2] GPS Date & Time: 2/11/2024 20:9:16 UTC
10 | [1] Setting date and time via GPS: 1707682156 $GNRMC
10 | [1] Time adjust 1707682146 secs

Still thinking through how to prioritize all of the different time sources (NTP, GPS, RTC, etc.) and how to manage them in a way so they don't fight against each other. Then will also adapt the code for other processor types such as ESP8266 and others, if I can.

Thinking we need a global variable or enum to keep track of what set the time and when the time was last set and check that every so often to validate that the time source is still working, and if not, fall back to the next accurate time source, if one is available. So GPS should be most accurate, and then NTP, and then RTC in my mind.

Then there's the question of what if the GPS receiver is not tied to the MQTT gateway but some other gateway down the line? How would that all work together if the MQTT gateway is getting its time source from NTP but another gateway is getting its time source from GPS then now do we get them to work together?

[timmbogner]

Cool cool. I'm still getting familiar with it, but getting there. It's kind of tempting to just stipulate that there may only be one time source per system, but that's no fun.

I think the scheme I had in mind before was that when each gateway receives the time, it repeats it to every single interface except the one it just received it from (source). It's taken for granted that all times are valid and they should only move away from their source. Between the source types, the only one with the possibility of becoming inaccurate is RTC. Make it so RTCs can be re-adjusted, but obviously GPS and MQTT can't. If an RTC is alone on a system, it will never get fixed, but if there is an accurate time being sent out from a GPS for example, that will eventually prevail.

<- That's just sort of a stream of consciousness recollection and not a definitive plan or idea btw. I think it makes sense but there are a lot of bases to cover. I still need to get a little more up to speed on things, but I may not be back around til tomorrow.

[cando-daves]

So glad your thinking/working on this and love the idea of the FDRS project.

RTC is really important for sensors on battery that only need to give data like once an hour or more. And a form of NTP to set it and keep it accurate. High precision is probably not generally important but I'm sure would be important in some cases. Maybe for those few cases they just use GPS.
I'm sure there will be cases where GPS signal is not available.
I've seen NTP discussed previously. I don't remember if it was regarding FDRS or another LORA project.
For low accuracy, within a couple seconds, (98% of all cases) I think push would work fine. But pull would still be needed for nodes that were deep sleeping when a push was sent. They wouldn't know a push was sent so would always have to pull when waking up.
But for high accuracy, pull would be better since the library would need to know the time for transmission and time for processing to calculate the required adjustment to compensate for those things.
I agree that you should be able to choose your source, GPS or a specific neighbor. Again, better for pull.
No point in frequent updates unless the device does not have RTC and loses the time while sleeping. For devices with RTC, once per day should be more than adequate, set in the config file.

One question when going for high accuracy is how to correct the time. NTP normally uses a nudge that makes the correction gradually. I kind of doubt that will be an option for most RTCs. My thought is to keep it simple and just set the time and report or log the delta.
I'm new to LORA stuff and don't know if it is possible to know the time from when a packet is received to when you start processing it and be sure when the packet will transmit. Seems like it would be necessary to be part of the LORA library to set a timestamp in the packet header at the last moment as it sends the response packet.

[aviateur17]

Hello David, hope you are doing well. GPS and RTC integration is already in the project. Of course you are welcome to improve on it. There is no time slewing (gradual time changes) built in. If a time difference is detected the time is just changed to the newly detected time.

GPS integration:

Farm-Data-Relay-System/src/fdrs_gateway_serial.h

Line 37 in d6e63eb

bool gpsParse(String input) {

NTP integration:

Farm-Data-Relay-System/src/fdrs_gateway_wifi.h

Line 192 in d6e63eb

// send an NTP request to the time server at the given address

RTC integration:

Farm-Data-Relay-System/src/fdrs_time.h

Line 75 in d6e63eb

#ifdef USE_RTC_DS3231

[timmbogner]

I'm going to make it my next priority to write up the documentation for timekeeping. I apologize that it's taken so long!
I'm planning to jump back in soon! Currently cleaning out my workshop in preparation...

[svdrummer]

Is this all becoming over complicated? Commercially. we use a date time variable. Behind the scenes a choice is made as to what updates those variables. GPS (with time offset) to update RTC if no lock then NTP (Careful with this. Poll to often and your IP is blocked)
No point is counting in milliseconds. If you go down that path, forget reliable comms, and in reality, why would FDRS need this.
GPS needs an "If time valid" check.
The original post asks about RTC vs NTP. Again, in commercial software, NTP is rarely used to poll. maybe once per day to update RTC.

[timmbogner]

This was a thread from earlier in the timekeeping development. It has since been completed and added, but there still aren't any instructions in the readmes. That will be my next task.
NTP is polled every 60 minutes by default, which hasn't given us any problems thus far. It is adjustable in config. I will look into the limits for our default time server. If it seems like a potential pitfall, I'll address this with a warning in the documentation.

I'd encourage you to take a look at how the timekeeping functionality turned out. I think we addressed some of these concerns in the finished product, but it'd be great to get some feedback!