Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add state pruner #161

Open
wants to merge 84 commits into
base: master
Choose a base branch
from
Open

feat: add state pruner #161

wants to merge 84 commits into from

Conversation

hal3e
Copy link
Contributor

@hal3e hal3e commented Nov 22, 2024

Adds the state_pruner service which will delete entries older that a certain retention duration from the db.

Additional refactoring done in this PR:

  • the old ports scheme was removed in favor of having each service declare its own ports and the adapters implementing them.
  • adapters moved to adapters folder
  • added test-helpers crate
  • moved service tests to integration tests

@segfault-magnet
bundle id now first requested from the db so that it may be
6f81493 
incorporated into the bundle
@segfault-magnet
progress on custom encoding decoding
e6f0b50
@segfault-magnet
blobs encoded are valid, decoding pending
0266aad
@segfault-magnet
wip
53f77d1
@segfault-magnet
tests passing
115583d
@segfault-magnet
tests passing
35af291
@segfault-magnet
Merge remote-tracking branch 'origin/master' into feat/blob_metadata
7372f62
@segfault-magnet
cleanup
be65f88
@segfault-magnet
cleanup of namespaces
d967ab7
@segfault-magnet
wip
b91bac1
@segfault-magnet
bundle versioning decoding and encoding
37fd9ce
@segfault-magnet
fixed default compression level
982b278
@segfault-magnet
integrated bundle encoder
aaf0de6
@segfault-magnet
tests passing
ab7a0f2
@segfault-magnet
cleanup
7eac67c
@segfault-magnet
fix typo
184cc5e
@segfault-magnet
move utils to encoding
27acdd0
@segfault-magnet
add feat flag
e54eba9
@segfault-magnet
improve debug output of bundle
692b416
@segfault-magnet
remove prints
78bb336
@segfault-magnet
fix gas usage
fbb8896
@segfault-magnet
reformat
06e7822
@segfault-magnet
add checks for empty blobs and for varying bundle ids
70ac7f8
@segfault-magnet
feature gate the kzg dep
512eeb6
@segfault-magnet
add kzg feature to eth blob dep
ac1f60a
@segfault-magnet
make header roundtrip test more robust
5098539
@segfault-magnet
move allocations to ingest method
b0d8c3c
@segfault-magnet
add tests for consuming exactly one blob doesnt allocate another
3128966
@segfault-magnet
add more checks for decoder
a72b0cb
@segfault-magnet
add blob with header to avoid tuples
217ea3b
@hal3e hal3e added the bug Something isn't working label Nov 22, 2024
@hal3e hal3e self-assigned this Nov 22, 2024
@hal3e hal3e added big and removed bug Something isn't working labels Nov 22, 2024
segfault-magnet
segfault-magnet reviewed Nov 30, 2024
View reviewed changes
committer/src/api.rs Show resolved Hide resolved
committer/src/setup.rs Show resolved Hide resolved
e2e/src/lib.rs Outdated Show resolved Hide resolved
packages/adapters/clock/src/lib.rs Outdated Show resolved Hide resolved
packages/adapters/storage/migrations/0007_cost_tracking.sql Outdated Show resolved Hide resolved
packages/adapters/storage/src/postgres.rs Outdated Show resolved Hide resolved
packages/adapters/storage/migrations/0007_cost_tracking.sql Show resolved Hide resolved
packages/adapters/storage/src/postgres.rs Show resolved Hide resolved
packages/services/src/state_pruner.rs Outdated Show resolved Hide resolved
packages/services/tests/state_pruner.rs Outdated Show resolved Hide resolved
segfault-magnet
segfault-magnet approved these changes Dec 5, 2024
View reviewed changes
Copy link
Contributor

@segfault-magnet segfault-magnet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ty for all the QoL improvements.

We should probably start off setting the pruning cutoff to something conservative like 2w.

@digorithm
Copy link
Member

Ty for all the QoL improvements.

We should probably start off setting the pruning cutoff to something conservative like 2w.

2w sounds like a safe starting point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@segfault-magnet segfault-magnet segfault-magnet approved these changes

@Salka1988 Salka1988 Salka1988 approved these changes

@digorithm digorithm Awaiting requested review from digorithm digorithm is a code owner

@MujkicA MujkicA Awaiting requested review from MujkicA MujkicA is a code owner

Assignees

@hal3e hal3e

Labels
big
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@hal3e @digorithm @Salka1988 @segfault-magnet