Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dev/robin/9780 veracity verify consistency command #24

Merged
merged 17 commits into from
Aug 30, 2024

Conversation

robinbryce
Copy link
Contributor

This adds first class support for local replication and verification of forestrie logs

The local reader implementation from veracity has been brought into the primary repositor so that it can benefit from more rigorous unit and integration testing.

The mechanism from the veracity implementation for scaning local directories for massifs has been extended:

  • It supports replicating the massif seals as well as the massifs
  • It supports replicating multiple logs under a single directory in a schema matching the remote forestrie blob paths.
  • It ensures the necessary verification and consistency checks are applied to the remote data before the data is replicated.
  • It can be used in a "one shot" mode or a continual processing mode. A rescan of the local cache directory can be forced

Additionally, it address the previous technical debt of having the azblob options polute the MassifReader interface

@henry739
Copy link
Contributor

Once this gets merged, will there be a follow up PR to remove the duplication from veracity and make it depend on the massifs package?

@honourfish
Copy link
Contributor

I agree, it would be nice to have the offline massif reader outside of veracity and in logverification or massifs

@robinbryce
Copy link
Contributor Author

Once this gets merged, will there be a follow up PR to remove the duplication from veracity and make it depend on the massifs package?

That is underway in parallel. datatrails/veracity#26

I don't intend to merge this until I/we are satisfied with the veracity end

@robinbryce robinbryce closed this Aug 27, 2024
@robinbryce robinbryce reopened this Aug 27, 2024
@robinbryce
Copy link
Contributor Author

closed by acident. ui confusion

Robin Bryce added 7 commits August 30, 2024 10:19
* refactor the local massif reader common paths to reduce some un-needed complexity
* remove an unused context from one interface
* some small comment nits
@robinbryce robinbryce requested a review from honourfish August 30, 2024 13:23
@robinbryce robinbryce merged commit eb7b313 into main Aug 30, 2024
1 check passed
@robinbryce robinbryce deleted the dev/robin/9780-veracity-verify-consistency-command branch August 30, 2024 13:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants