[DONT_MERGE] Rocksdb storage #527
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* methods use atomic write/read batches, remove flush * update test to teardown and use async commit * tree.get can be passed batch * add getByteRange helper * proof helpers all support batches * remove stale code * update to new storage api * update to use peakLastTreeNode * use deleteRange for truncations * reopen a tree from db storage * remove cache * right span available nodes * skip cache test * skip all but merkle tree tests * update for batch api * rename to dbBatch for clarity * standard * kill stale code * rename to read/writeBatch * truncation should delete parent nodes
* use rocks db storage for bitfield pages * update tests to use rocksdb storage * rocksdb returns buffer with length * enable bitfield tests
* remove oplog and load header from rocks * move block store to rocksdb * remove oplog entries * create write batches and pass to persisting methods * comment out copyPrologue and insertBatch for now * enable lib/core test * write header info directly to batch * batch updates and flush to onupdate later * update user data api * userData key is string in storage * merkle tree does not own storage * merkle tree write operations take in batch * block operations are sync * bitfield flush is sync and takes write batch * pass db directly to Core.open * set user data operates on a batch * bitfield does not ref storage
* update index.js to use new core api * encode manifest when storing * tweak basic tests * flush reader after caching * tryFlush --------- Co-authored-by: Mathias Buus <[email protected]>
* update index.js to use new core api * encode manifest when storing * tweak basic tests * use opts.discoveryKey if passed * merkle tree proof settled externally * enable basic tests * replicator creates batch and fulfills request separately * ensure verify ops are using top level write batches * bugfix: tree indexes should be multiplied by 2 * always return tree proof and await handleRequests * update replicate tests * always return proof request * clear blocks now takes start and end explicitly * update lib/multisig with new proof api * flush manifest to batch during append * standard * update tests and enable as many as possible * only finalise batch after write is flushed * persist correct value for fork * tidy up tests * fix top level user data api * update snapshot tests * finalise reorg batch * standard * reorg batch is not finalised * only clear parent nodes if ancestors > 0 * only upgrade batches need be finalised * verifyBatchUpgrade optionally does not write * tidy up todos * always write manifest if we can * fixup * enable cache tests * assert that merkle tree is committed before finalising * ensure stream is drained before handling requests * missing new keyword * add todo for critical bugfix * store user updates on write batch * store pending bit sets until after flush * use sorted list of intervals instead of map * verify methods create their own batches * allow updateContig to check dirty bitfield * only one batch is allowed to be active at any time * await flushing before processing get * add @mafintosh interval tracker * alloc and set using uintarray * reenable bitfield tests * tidy update flow and redirect for easier refactor * add BitInterlude class * bit interlude wraps bitfield and generates pages * use bit interlude to calculate contig length * optimise setting bitfield buffers * use quickbit in bit interlude * move all batch and flush logic into handle request * add initial update abstraction * extend usage of update * move clear and clearBatch to update * contiguous length is drop dependent * remove last methods using write batch directly * tidy up * no longer need to mark pages as dirty * remove dead props * flushed returns if no active batch
* implement copyPrologue * standard fixes * review by @mafintosh
* define scoped states per session * return byteLength from updated batch * pass length to createNamedSession * add tree method for merging one tree into another * add core method for committing state * state stores treeLength * session gets from state * handle case when no tree nodes are added * fix session get and truncate * core truncate passes entire state down * batch has all info after being reconciled * update batch tests * session uses state props * pass whole batch to be flushed * copy over bitfield to named session * minor corrections * update batch tests to be on named sessions * update core tests * make sure we pass state to createUpdate * add checkout option for batches * optionally pass discovery key to core * each state has an independent mutex * bitfield should full copy entire buffer * user data is available sync for default state * named session is always writable * commit can specify length * corestore owns storage * update flushed length prop * expose restore batch on main class * encryption should use session state * rebase main * fix storage usage * test core length is increasing * standard fixes * use state props and pass tree to multisig * fix bitfield page copy * update tests * move to session state to dedicated abstraction * pass parent state as capability * ensure we have correct treeLength and up to date bitfield when opening state * only write each bitfield page once per flush * truncate and clear mutate treeLength * fixes for batch tests * enable batch tests * overwrite if session when opts.refresh is set * enable batch clear test * close storage when we close the core * storage closes automatically * auto teardown any created core * we have to pass an error to mutex destruction... * close all test cores * more missing close * more closes * missing session close * close db in createIfMissing test * more closes * make sure all sessions are closed too * checkout should only truncate named session * state tracks active sessions on open and close * screen for failing test * core closes state * close existing state when creating named session * missing session close * more closes * pass runner to helper * close core instead * close state first and fix teardown order * close state last * missing close * missing close * missing close --------- Co-authored-by: Mathias Buus <[email protected]>
* wip * adjust tests * truncate issues a deletion of the range also * start + 1 -> end
* rename clone/close to ref/unref * add snapshot method to session state * snapshots are not writable * snapshotted session takes state snapshot * fall back to core if snapshot does not have data * review by @mafintosh * gc snapshot * fix snapshot teardown * _snapshot -> storageSnapshot * dry it a bit --------- Co-authored-by: Mathias Buus <[email protected]>
chm-diederichs
force-pushed
the
rocksdb
branch
from
e9b2162 to
6c213d7
Compare
--------- Co-authored-by: HDegroote <[email protected]> Co-authored-by: Mathias Buus <[email protected]> Co-authored-by: rafapaezbas <[email protected]>
* add failing test * fix the active counter with the fix from rocks branch * do not allow core teardown during peer attachment if sessioned * can just bump sessions instead of the extra flag * move peer session flow fully to the makepeer lifecycle
* block reqs are not readded while being processed * check we do not have block locally before requesting * check bitfield before deciding to wait * add comment --------- Co-authored-by: Mathias Buus <[email protected]>
* wip * checkpoint * tests pass now * add safe close api on core and move replicator there * more stuff in core * all hooks live in core * inline replicator * premerge * no need to skip * add static helper for making core cores * bye bye non-sparse mode * Core manages replicator lifecycle (#592) * state unref is sync * core closes storage * replicator no longer needs session * remove force close from session * fix bad condition * core being set is invariant * remove stale method * query sessions on underlying core * move autoClose onto Core * rename close to destroy * core destroys state * preload is now simply a promise * preload promise can return opts * no need to check if state is active * add onidle hook * idle waits for mutex to be free also * remove from option * exclusive sessions * always emit close * fix test * pass userdata to create for atomicity * core always refs default state * session explicitly refs state --------- Co-authored-by: Christophe Diederichs <[email protected]> Co-authored-by: Christophe Diederichs <[email protected]>
* add memory overlay test * fix put * implement memory overlay block deletion * add memory overlay block deletion tests * move tip list to abstraction and add tests * treeNodes use tip list * move memory overlay to hypercore-on-the-rocks * rename peak to peek
* commit acquires lock directly * check conditions after acquiring lock * update dependency in place and delete blocks * update dependency on truncate and clear if necessary * move memory overlay to hypercore-on-the-rocks * rename peak to peek * use dependencies to compute flushed length * bit-interlude supports multiple deletions * add explicit state overwrite method * memory overlay always has flushedLength as -1 * signature is verified against committed tree * use flat-tree patch method * review from @mafintosh * only one call to splice
* change bitfield setRange api to start + end * update usage of remote bitfield
* easy ones * do not rely on events to adopt encryption compat * snapshot compat length lives on the session now * wip * last stretch * remove dead code
* BitInterlude does not need underlying bitfield * factor out bitfield from session state * update tests * update audit to use batch directly * optimise hot paths in has * updateContigBatch returns -1 as default * review by @mafintosh
* move createSession onto state * need to special case draft state * do not throw error if we are closing
* expose user data stream from lib/core.js * enclose ternary condition * add user data stream tests * rename to createUserDataStream
* add tree snapshot * snapshots store explicit tree snapshot * add snapshotFrom method * session update does not affect snapshot state * refactor snapshot update test
* allow root session to define session list * review by @chm-diederichs
* add test for snapshot on named batch * move dependency mutation to storage api
* inherit more from parent * simplify
* drafts are writable * session overwrite should emit events
* add initial moveTo API * snapshot makes full tree clone * add test for snapshots * parent state may be swapped during ready * session has method for transferring * only check compat if block is not local * add snapshot move to checks * retrying requests on session move * transfer all replicating io when moving sessions * drafts are writable * session overwrite should emit events * add option for overwrite to update dependency * do not create snapshot over existing snapshot * ref session state in constructor * fix non writable named sessions * emit migrate event on moveTo --------- Co-authored-by: Mathias Buus <[email protected]>
* optionally pass atomizer to append * commit and overwrite accept atomizer * commit errors on fail and destroys atomizer * flushWriteBatch always uses active batch * add test for overwrite * setUserData accepts atomizer * throw error after destroying atomizer * standard test * refactor mutex to return lock * atomizer acquires lock * add tests with concurrent operations * fix test condition * rewrite tests to be on same storage * enable atomic tests * rename atomizer to atom * moveTo supports atomicity * moveTo acquires lock
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Opening the main PR so CI can run explicitly and we can follow