Dev/jgough/9467 mmr massifs move #6
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Remove published code & refactor * put all of massifcontext into the public repo. It is important for maintainability that lives in one place. * delete all the code that is now public and update services accordingly AB#9017 * update go mod for go-datatrails-merklelog 0.0.2 * remove assetisms from mmr * Move packages back from go-datatrails-merklelog * Restores the packages that were moved to go-datatrails-merklelog. * Adds them back as packages with independent go.mod's * Contains significant work in flight towards 8336 AB#9053 AB#8336 * bump go-datatrails-common * linter fixups and remove accidentaly commited test dir * fixup dockerfiles * go mod tidy --------- Co-authored-by: Robin Bryce <[email protected]>
* MassifContext can be used with GetRoot This PR adds the CreatePeaksStackMap method to the MassifContext. The default Add method on the MassifContext is optimised for the Add leaf case. As this is the hot path. GetRoot accesses the mmr differently, so when the context is used for GetRoot it needs a more explicit picture of the peaks carried over from previous massifs The CreatePeaksStackMap method should be called before using the context with GetRoot. The MassifReader type has gained With option support to it's New method so that it can be configured easily for this case. AB#8336 * Signed confirmation of log state * implements a logconfirmer which periodically verifies the consistency of the log and signs the latest root. * Every massif backing the log is associated with a single 'Signed Tree Head'. While the massif is being filled, its just the most recent head. Once the massif is complete, the logconfirmer will sign the root corresponding to the MMRSize at the end of that massif. There after, that published signed tree head will never change. * Also, rename triecommon -> merklelog AB# 8336 * linter issues * Review feedback & fix issue with mapped peak stack * remove duplicated logging --------- Co-authored-by: Robin Bryce <[email protected]>
* Provide a polling tenant log confirmer This uses much of the work towards the priority queue based implementation, but should be easier to get through initial review. The priority queue based version couldn't horizontally scale without a redis based queue. And that is just to much for one hit See https://github.com/datatrails/forestrie/tree/dev/robin/8336-signed-merkle-roots-5-proto-scheduler AB#8336 Also, * fix a peak stack issue, and make it less confusing * add a fairness tes * rationalise tests with respect to thread safety * commit to last snowflake id for mmr size in root signature. so that last activity can be based off the actual time of the activity, rather than the last time the signature was produced. Also, this gives us a continuous and unique value for the log from which we can index configuration changes independently of the massif blobs. * Use the generator idState instance in GenerateTenantLog * minor self review cleanups * fix merge conflict and minor pr fixup --------- Co-authored-by: Robin Bryce <[email protected]>
New rune 'Task modules' which ensures that all go.mod files are up-to-date and correct. All go.mod files are also updated to be correct. Additional 'replace' directves to ensure that go mod download does not require authorization for upstream. AB#9031
Use go 1.22.1 Uograde golangci-lint 1.56.2 Upgrading golangci-lint simultaneously is usually required. golangci-lint is go 1.22 compatible from 1.56.0 onwards. AB#9064
* defer reading blobs and verifying seals until we know the log has progressed The fundamental cause of AB#9236 was egregious ineficiency. It was taking many seconds, some times 10's of seconds to process the tenants that *might* need confirming. This change re-arrangest things so we defer the massif reading entirely until we know the log has progressed. We do this by relying on the content length and the last modified properties we get from azure blob storage. At worst, failure simply causes us not to seal the log. It can not result in corruption or mis signing. Before actually creating a new seal we, as before, verify the current seal and verify the new log state is consistent with the verified previous state. The methods for navigating the mmrblob basic format were previously only available on the MassifContext. To facilitate checking the number of entries in the log based only on the content length those have been re-factored as stand alone functions which now live in the logformat.go file Also renamed from logentry.go to logformat.go In summary, * Use the last modified on the blob meta data to decide if the confirmation interval has elapsed rather than reading the whole massif of every tenant regardless. * Use the content length of the masssif to decide if more entries have been added since the last seal was made * Make the fundamental mmrblob format functions available as module scoped functions. * Defer verification of the seal signature until *just* after the length check. This fails safe and is measurably a lot more efficient. * Re-organise the pending tenant checks so we don't read the massif's and seals multiple times from blob storage restore cheatsheet bad merge fixup linter issues fixup tests * use lazy context rather than the more obscure lazy context * review updates * logical context -> lazy context * remove the TODO: add tests that are already filled in --------- Co-authored-by: Robin Bryce <[email protected]>
Also takes latest key each time we sign, owing to new signer factory pattern. Added system test to check format of KID in header. re AB#9185 Co-authored-by: Henry Jewell <[email protected]>
* Rename forestrie mmrblobs to massifs re: AB#9461 --------- Co-authored-by: jgough <[email protected]>
* identify the latest blobs for each tenant Please note this is work in progress. The logwatcher_*.go files will move once the repostitory shuffling is done AB#9402 * review: consistenly provide function comments * Update go-forestrie/demos/veracity/logwatcher_pathparse.go Co-authored-by: Joe Gough <[email protected]> Signed-off-by: robinbryce <[email protected]> --------- Signed-off-by: robinbryce <[email protected]> Co-authored-by: Robin Bryce <[email protected]> Co-authored-by: Joe Gough <[email protected]>
AB#9402 A service to monitor the recently active tenant massifs and seals. It will be responsible for posting batches to a topic. The sealer will pick up the batches, with the assumption that the batch is *at least* those tenants active since the last batch. But which MAY include redundant entries. In cluster tests show the idle query costs < 5m. And the CPU cost, both empirically and from the azure docs, should be fairly constant, but no worse than proportional to the number of *matched blobs* The azure request cost is 1. And we issue the query once per service instance per deployment per interval. The interval defaults to 2 seconds. We could probably make that faster. THe time horizon under consideration is currently set to 30 seconds. The azure docs suggest that is safe. But super safe would be about a minute or two. Other notables: snowflakeid was causing un-necessary dependencies on datatrails event api consuming code. And it was particularly painful for this change. So I've provisionaly refactored and moved to go-datatrails/massifs/snowflakeid task helm:test:forestrie was broken It now correctly infers the --values files and renders the forestrie chart corectly The only remaining gap with respect to skaffold.yml is how skaffold images are specified review: minors and and rootpublisher -> batchpublisher Co-authored-by: Robin Bryce <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.