You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently (or more accurately, once #2538 merges), we expect shared sstables to have a predefined target level that must be either 5 or 6. We use this to give those sstables a reserved sequence number. However we will need to update the ingestion path to relax this and allow for shared sstables to come in layers deeper than 2 levels, as progressive restore is expected to throw multiple layers of overlapping SSTs at Pebble at once. This will happen in the form of successive Ingest (no excise) calls, where each call assigns an increasing sequence number to ingested sstables and determines target levels just like non-shared tables. Note that we don't need to do an excise at the same time as these ingest calls, and we also don't need to ensure that these files will fit within the excise span (as there is none).
This new Ingest call will also accept prefix rewrite rules (see #2595) and bucket info (see #2594).
I think we also need some terminology improvement, that will make the ingest interface clearer in terms of what the caller is trying to do. Perhaps we should reserve the (existing) term "foreign sst" to refer to an sst that is from a "foreign Pebble" instance. The ingestion of such ssts only makes sense together with other state from that foreign Pebble instance that will layer on top of these foreign ssts. We could also call this a "foreign ingest". Then these rules about target level 5 and 6 only apply to these foreign ingests. And regular ingests can also use ssts in shared storage.
Currently (or more accurately, once #2538 merges), we expect shared sstables to have a predefined target level that must be either 5 or 6. We use this to give those sstables a reserved sequence number. However we will need to update the ingestion path to relax this and allow for shared sstables to come in layers deeper than 2 levels, as progressive restore is expected to throw multiple layers of overlapping SSTs at Pebble at once. This will happen in the form of successive
Ingest
(no excise) calls, where each call assigns an increasing sequence number to ingested sstables and determines target levels just like non-shared tables. Note that we don't need to do an excise at the same time as these ingest calls, and we also don't need to ensure that these files will fit within the excise span (as there is none).This new
Ingest
call will also accept prefix rewrite rules (see #2595) and bucket info (see #2594).Jira issue: PEBBLE-48
Epic CRDB-40359
The text was updated successfully, but these errors were encountered: