Schema + versioning concept for IoText data protocol

[bieli]

In general with the schema-less basic version of the protocol we can send data like this:

t|3900237526042,d|device_name_001,m|val_water_001=i:1234,m|val_water_002=i:15,m|bulb_state=b:1,m|connector_state=b:0,m|temp_01=d:34.4,m|temp_02=d:36.4,m|temp_03=d:10.4,m|pwr=d:12.231,m|current=d:1.429,m|current_battery=d:1.548

It will be more optimal - and of course it is prelude to Change Data Capture technique - to add an implementation of schema versioning with indexes instead of repeated metrics names like this:

s|v1,t|3900237526042,d|device_name_001,001=1234,002=15,003=1,004=0,005=34.4,006=36.4,007=10.4,008=12.231,009=1.429,00a=1.548

as you can see, we have a special item s|v1 - information about versioning and version name.

At the same time schema message needs to be sent to the server/receiver, too - schema format proposal:

S|v1,001|val_water_001=i,002|val_water_002=i,003|bulb_state=b,004|connector_state=b|temp_01=d,006|temp_02=d,007|temp_03=d,008|pwr=d,009|current=d,00a|current_battery=d

as you can see, we have a bigger S item code/kind for schema definition and all repeated metrics names are indexed by hexadecimal 3 positional digits. It gives us ~4k metrics in one row/message.

With schema + versioning, we can save approx. 50% of message size. Much more, compared to JSON formated the same message!

>>> len0 = len('t|3900237526042,d|device_name_001,m|val_water_001=i:1234,m|val_water_002=i:15,m|bulb_state=b:1,m|connector_state=b:0,m|temp_01=d:34.4,m|temp_02=d:36.4,m|temp_03=d:10.4,m|pwr=d:12.231,m|current=d:1.429,m|current_battery=d:1.548')
>>> len1 = len('s|v1,t|3900237526042,d|device_name_001,001=1234,002=15,003=1,004=0,005=34.4,006=36.4,007=10.4,008=12.231,009=1.429,00a=1.548')
>>> len0
226
>>> len1
124
>>> 

>>> len1 / len0
0.5486725663716814

>>> len0_json = len('[{"t":3900237526042},{"d":"device_name_001"},{"m":[{"val_water_001":{"i":1234}},{"val_water_002":{"i":15}},{"bulb_state":{"b":1}},{"connector_state":{"b":0}},{"temp_01":{"d":"34.4"}},{"temp_02":{"d":"36.4"}},{"temp_03":{"d":"10.4"}},{"pwr":{"d":"12.231"}},{"current":{"d":"1.429"}},{"current_battery": {"d":"1.548"}}]}]')
>>> len0_json
319