Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optimization for TEXT fields with RLE algo. (https://en.wikipedia.org/wiki/Run-length_encoding) - new sign for TEXT field upper 'T' letter #22

Open
bieli opened this issue Jan 11, 2024 · 1 comment
Labels
enhancement New feature or request

Comments

@bieli
Copy link
Owner

bieli commented Jan 11, 2024

RLE is a common algo. and probably nothing special, but in case of data streams with TEXT-type data, we can use RLE to optimize bytes used to transmit data records.

Example text:

WWWWWWWWWWWWBWWWWWWWWWWWWBBBWWWWWWWWWWWWWWWWWWWWWWWWBWWWWWWWWWWWWWW

With an RLE data compression algorithm we can see:

12W1B12W3B24W1B14W

We need to implement a special char in IoText data protocol for this - "T" high letter as RLE compressed data identifier.
Example of record in IoText data protocol with suggested field type:

t|3900267526042,d|meter_power_01,m|err_msg=T:12W1B12W3B24W1B14W

in a typical TEXT field type, we will have the below message:

t|3900267526042,d|meter_power_01,m|err_msg=t:WWWWWWWWWWWWBWWWWWWWWWWWWBBBWWWWWWWWWWWWWWWWWWWWWWWWBWWWWWWWWWWWWWW

Little optimizations are not too bad, if we fighting with bytes in the transmission channel - for example in Narrow Band IoT or other thin options/transmission channels. Power consumption, it's a strongly connected aspect for battery-powered IoT devices.

@bieli bieli added the enhancement New feature or request label Jan 11, 2024
@bieli
Copy link
Owner Author

bieli commented Apr 3, 2024

"T" upper letter is reserved for this implemention: #24 - try special sign from UTF-8 maybe... է

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant