Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ingest: Allow ingestion of overlapping shared files #2596

Open
itsbilal opened this issue Jun 5, 2023 · 1 comment
Open

ingest: Allow ingestion of overlapping shared files #2596

itsbilal opened this issue Jun 5, 2023 · 1 comment

Comments

@itsbilal
Copy link
Contributor

itsbilal commented Jun 5, 2023

Currently (or more accurately, once #2538 merges), we expect shared sstables to have a predefined target level that must be either 5 or 6. We use this to give those sstables a reserved sequence number. However we will need to update the ingestion path to relax this and allow for shared sstables to come in layers deeper than 2 levels, as progressive restore is expected to throw multiple layers of overlapping SSTs at Pebble at once. This will happen in the form of successive Ingest (no excise) calls, where each call assigns an increasing sequence number to ingested sstables and determines target levels just like non-shared tables. Note that we don't need to do an excise at the same time as these ingest calls, and we also don't need to ensure that these files will fit within the excise span (as there is none).

This new Ingest call will also accept prefix rewrite rules (see #2595) and bucket info (see #2594).

Jira issue: PEBBLE-48

Epic CRDB-40359

@sumeerbhola
Copy link
Collaborator

I think we also need some terminology improvement, that will make the ingest interface clearer in terms of what the caller is trying to do. Perhaps we should reserve the (existing) term "foreign sst" to refer to an sst that is from a "foreign Pebble" instance. The ingestion of such ssts only makes sense together with other state from that foreign Pebble instance that will layer on top of these foreign ssts. We could also call this a "foreign ingest". Then these rules about target level 5 and 6 only apply to these foreign ingests. And regular ingests can also use ssts in shared storage.

@jbowens jbowens moved this to Disagg - Online restore in [Deprecated] Storage Jun 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Status: Disagg - Online restore
Development

No branches or pull requests

2 participants