Critial Time recordings might be significatly wrong due to clock synchronization issues

[tmasternak]

Overview

The CriticalTime for a message is calculated in the processing endpoint partially based on the time header set by the sending endpoint.

If system clocks on the sender and processing endpoint differ significatly the recordings for CT might be considerably off. E..g it's possible for CT value to me smaller than Processing Time or even negative.

Possible solution

Detect when time value in the header for recieved message is smaller than the curren system time on the processing endpoint and log WARN.

/cc: @ramonsmits

[Scooletz]

Would a small effort approach like the following:

criticalTime = max(criticalTime,processingTime)

would be sufficient enough @Particular/metrics-maintainers ?

[dvdstelt]

Definitely

[dvdstelt]

Issue though is that if clock offset is large enough, not a single metric will have a criticalTime that is higher than processingTime.
Should we log a message (maybe every x-th measurement) that this is occurring? Or at the very least document it. Because people will start asking questions about this.

[ramonsmits]

@dvdstelt That is what I suggested on Slack, that if that happens that we log an event. So I'm all for that to not hide it and give guidance on how the customer can resolve it. I also raised https://github.com/Particular/PlatformDevelopment/issues/1839 to make system clock difference appear as errors in ServicePulse.

[mikeminutillo]

I definitely prefer https://github.com/Particular/PlatformDevelopment/issues/1839 to putting a floor on Critical Time. If we adjust Critical Time that way we are potentially hiding a problem from the user. That said, if the clocks drift the other way we don't have an easy way to detect it. Is this something we should be solving?

[ramonsmits]

Is this something we should be solving?

@mikeminutillo Maybe not solve, but we should alert/indicate this state (time sent>current time).