pkg: add FlashDB

[benpicco]

Contribution description

This adds a package for FlashDB, an embedded time series and key-value data base.

It can either operate directly on a raw MTD device (flashdb_mtd) or use the VFS layer for file based operation.

Testing procedure

I added a test that uses the examples provided by FlashDB.

tests/pkg_flashdb

2022-02-05 00:21:09,418 # main(): This is RIOT! (Version: 2022.04-devel-117-gda35d-pkg/flashdb)
2022-02-05 00:21:09,591 # ==================== kvdb_basic_sample ====================
2022-02-05 00:21:09,601 # get the 'boot_count' value is 1
2022-02-05 00:21:11,210 # set the 'boot_count' value to 2
2022-02-05 00:21:11,215 # ===========================================================
2022-02-05 00:21:11,220 # ================= kvdb_type_string_sample =================
2022-02-05 00:21:12,636 # create the 'temp' string KV, value is: 36C
2022-02-05 00:21:12,642 # get the 'temp' value is: 36C
2022-02-05 00:21:14,057 # set 'temp' value to 38C
2022-02-05 00:21:14,340 # delete the 'temp' finish
2022-02-05 00:21:14,346 # ===========================================================
2022-02-05 00:21:14,351 # ================== kvdb_type_blob_sample ==================
2022-02-05 00:21:15,241 # create the 'temp' blob KV, value is: 36
2022-02-05 00:21:15,272 # get the 'temp' value is: 36
2022-02-05 00:21:16,689 # set 'temp' value to 38
2022-02-05 00:21:16,972 # delete the 'temp' finish
2022-02-05 00:21:16,977 # ===========================================================
2022-02-05 00:21:17,013 # ======================= tsdb_sample =======================
2022-02-05 00:21:17,017 # append the new status.temp (36) and status.humi (85)
2022-02-05 00:21:17,022 # append the new status.temp (38) and status.humi (90)
2022-02-05 00:21:17,028 # [query_cb] queried a TSL: time: 19, temp: 36, humi: 85
2022-02-05 00:21:17,034 # [query_cb] queried a TSL: time: 21, temp: 38, humi: 90
2022-02-05 00:21:17,060 # [query_by_time_cb] queried a TSL: time: 19, temp: 36, humi: 85
2022-02-05 00:21:17,066 # [query_by_time_cb] queried a TSL: time: 21, temp: 38, humi: 90
2022-02-05 00:21:17,084 # query count is: 0
2022-02-05 00:21:17,093 # set the TSL (time 19) status from 3 to 3
2022-02-05 00:21:17,374 # set the TSL (time 21) status from 3 to 3
2022-02-05 00:21:17,673 # ===========================================================

tests/pkg_flashdb_mtd

2022-02-05 00:12:48,097 # main(): This is RIOT! (Version: 2022.04-devel-118-g80d9a-pkg/flashdb)
2022-02-05 00:12:48,104 # [I/FAL] Flash Abstraction Layer (V0.5.0) initialize success.
2022-02-05 00:12:48,117 # ==================== kvdb_basic_sample ====================
2022-02-05 00:12:48,120 # get the 'boot_count' value is 12
2022-02-05 00:12:48,132 # set the 'boot_count' value to 13
2022-02-05 00:12:48,137 # ===========================================================
2022-02-05 00:12:48,143 # ================= kvdb_type_string_sample =================
2022-02-05 00:12:48,152 # create the 'temp' string KV, value is: 36C
2022-02-05 00:12:48,155 # get the 'temp' value is: 36C
2022-02-05 00:12:48,157 # set 'temp' value to 38C
2022-02-05 00:12:48,160 # delete the 'temp' finish
2022-02-05 00:12:48,165 # ===========================================================
2022-02-05 00:12:48,170 # ================== kvdb_type_blob_sample ==================
2022-02-05 00:12:48,180 # create the 'temp' blob KV, value is: 36
2022-02-05 00:12:48,182 # get the 'temp' value is: 36
2022-02-05 00:12:48,185 # set 'temp' value to 38
2022-02-05 00:12:48,187 # delete the 'temp' finish
2022-02-05 00:12:48,192 # ===========================================================
2022-02-05 00:12:48,198 # ======================= tsdb_sample =======================
2022-02-05 00:12:48,203 # append the new status.temp (36) and status.humi (85)
2022-02-05 00:12:48,207 # append the new status.temp (38) and status.humi (90)
2022-02-05 00:12:48,212 # [query_cb] queried a TSL: time: 1, temp: 36, humi: 85
2022-02-05 00:12:48,217 # [query_cb] queried a TSL: time: 1, temp: 38, humi: 90
2022-02-05 00:12:48,222 # [query_by_time_cb] queried a TSL: time: 1, temp: 36, humi: 85
2022-02-05 00:12:48,228 # [query_by_time_cb] queried a TSL: time: 1, temp: 38, humi: 90
2022-02-05 00:12:48,229 # query count is: 0
2022-02-05 00:12:48,233 # set the TSL (time 1) status from 3 to 3
2022-02-05 00:12:48,236 # set the TSL (time 1) status from 3 to 3
2022-02-05 00:12:48,242 # ===========================================================

Issues/PRs references

includes #17341 for convenience
repo of FlashDB

[maribu]

I like this one :) The CI has some comments.

[fjmolinas]

One question out of curiosity, the GitHub page sys its supports wear balance, do you know how that is done? I was trying to find some information to estimate flash wear out when using the API but didn't find a section on that immediately.

[kfessel]

There is no reason to further hold this back (review stages 1 to 5 are already green by kaspar) i did run the test on native and read this.

Issues raised earlier are addressed.

Please squash and activate CI.

I would like to see the pseudomodules listed in Doxygen.
I think there should be some kind of warning that this will not do wear leveling and advising a solution like littlefs2.
I also would like to see the title of the Doxygen group changed a little for more userfullnes of that.

Please squash these changes right in.

[benpicco]

I think there should be some kind of warning that this will not do wear leveling and advising a solution like littlefs2.

Whaat? I thought that was one of it's selling points.

[kfessel]

I think there should be some kind of warning that this will not do wear leveling and advising a solution like littlefs2.

Whaat? I thought that was one of it's selling points.

maybe i missed something or the wearleveling is done another way than in littlefs2.
The memory-file for (mtd) on native has one instance of the datebase after multiple runs.
The littlefs2 memoryfile (vfs version) has the database multiple times (every time written to another place)
Maybe this does not level wear across reboot?

[kfessel]

seems like the compactness of the written data (littlefs2 is more spacey when writing to different places) in the mtd threw me off -> it writes to different places (levels the wear)

[riot-ci]

Murdock results

✔️ PASSED

674e3ad tests/pkg_flashdb: add test for FlashDB

Success	Failures	Total	Runtime
6887	0	6888	09m:31s

Artifacts

• Documentation preview

[kfessel]

I had another look at the writes (native mtd debug)-> only writes are happening no erases (i assume until rollover)

There is a huge number of small writes (sometimes to the same location but no erase) the test has a increasing number of writes with every run as the database increases with each run by some tvdb entries. I think this is ok.

grep write 
mtd_native: write from page 4, offset 0xa3 count 1
mtd_native: write from page 5, offset 0x47 count 1
mtd_native: write from page 5, offset 0x4b count 20
mtd_native: write from page 5, offset 0x5f count 10
mtd_native: write from page 5, offset 0x69 count 4
mtd_native: write from page 5, offset 0x47 count 1
mtd_native: write from page 4, offset 0xa3 count 1
mtd_native: write from page 5, offset 0x6d count 1
mtd_native: write from page 5, offset 0x71 count 20
mtd_native: write from page 5, offset 0x85 count 4
mtd_native: write from page 5, offset 0x89 count 3
mtd_native: write from page 5, offset 0x6d count 1
mtd_native: write from page 5, offset 0x6d count 1
mtd_native: write from page 5, offset 0x8c count 1
mtd_native: write from page 5, offset 0x90 count 20
mtd_native: write from page 5, offset 0xa4 count 4
mtd_native: write from page 5, offset 0xa8 count 3
mtd_native: write from page 5, offset 0x8c count 1
mtd_native: write from page 5, offset 0x6d count 1
mtd_native: write from page 5, offset 0x8c count 1
mtd_native: write from page 5, offset 0xab count 1
mtd_native: write from page 5, offset 0xaf count 20
mtd_native: write from page 5, offset 0xc3 count 4
mtd_native: write from page 5, offset 0xc7 count 4
mtd_native: write from page 5, offset 0xab count 1
mtd_native: write from page 5, offset 0xab count 1
mtd_native: write from page 5, offset 0xcb count 1
mtd_native: write from page 5, offset 0xcf count 20
mtd_native: write from page 5, offset 0xe3 count 4
mtd_native: write from page 5, offset 0xe7 count 4
mtd_native: write from page 5, offset 0xcb count 1
mtd_native: write from page 5, offset 0xab count 1
mtd_native: write from page 5, offset 0xcb count 1
mtd_native: write from page 21, offset 0x8 count 1
mtd_native: write from page 21, offset 0xc count 12
mtd_native: write from page 2f, offset 0x88 count 8
mtd_native: write from page 21, offset 0x8 count 1
mtd_native: write from page 21, offset 0x18 count 1
mtd_native: write from page 21, offset 0x1c count 12
mtd_native: write from page 2f, offset 0x80 count 8
mtd_native: write from page 21, offset 0x18 count 1
mtd_native: write from page 20, offset 0x28 count 1
mtd_native: write from page 20, offset 0x38 count 1
mtd_native: write from page 20, offset 0x48 count 1
mtd_native: write from page 20, offset 0x58 count 1
mtd_native: write from page 20, offset 0x68 count 1
mtd_native: write from page 20, offset 0x78 count 1
mtd_native: write from page 20, offset 0x88 count 1
mtd_native: write from page 20, offset 0x98 count 1
mtd_native: write from page 20, offset 0xa8 count 1
mtd_native: write from page 20, offset 0xb8 count 1
mtd_native: write from page 20, offset 0xc8 count 1
mtd_native: write from page 20, offset 0xd8 count 1
mtd_native: write from page 20, offset 0xe8 count 1
mtd_native: write from page 20, offset 0xf8 count 1
mtd_native: write from page 21, offset 0x8 count 1
mtd_native: write from page 21, offset 0x18 count 1

[kfessel]

Squash and Go

[benpicco]

Thank you for the review!

bors merge

[bors]

Build failed (retrying...):

• Murdock

[bors]

Canceled.

[benpicco]

bors merge

[bors]

🕐 Waiting for PR status (GitHub check) to be set, probably by CI. Bors will automatically try to run when all required PR statuses are set.

[benpicco]

bors merge

[bors]

Build succeeded:

• Murdock
• static-tests
• tools-build-success