-
Notifications
You must be signed in to change notification settings - Fork 494
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
## Problem One doesn't know at tenant creation time how large the tenant will grow. We need to be able to dynamically adjust the shard count at runtime. This is implemented as "splitting" of shards into smaller child shards, which cover a subset of the keyspace that the parent covered. Refer to RFC: #6358 Part of epic: #6278 ## Summary of changes This PR implements the happy path (does not cleanly recover from a crash mid-split, although won't lose any data), without any optimizations (e.g. child shards re-download their own copies of layers that the parent shard already had on local disk) - Add `/v1/tenant/:tenant_shard_id/shard_split` API to pageserver: this copies the shard's index to the child shards' paths, instantiates child `Tenant` object, and tears down parent `Tenant` object. - Add `splitting` column to `tenant_shards` table. This is written into an existing migration because we haven't deployed yet, so don't need to cleanly upgrade. - Add `/control/v1/tenant/:tenant_id/shard_split` API to attachment_service, - Add `test_sharding_split_smoke` test. This covers the happy path: future PRs will add tests that exercise failure cases.
- Loading branch information
Showing
19 changed files
with
1,088 additions
and
24 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
46 changes: 46 additions & 0 deletions
46
control_plane/attachment_service/src/persistence/split_state.rs
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,46 @@ | ||
use diesel::pg::{Pg, PgValue}; | ||
use diesel::{ | ||
deserialize::FromSql, deserialize::FromSqlRow, expression::AsExpression, serialize::ToSql, | ||
sql_types::Int2, | ||
}; | ||
use serde::{Deserialize, Serialize}; | ||
|
||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash, PartialOrd, Ord, FromSqlRow, AsExpression)] | ||
#[diesel(sql_type = SplitStateSQLRepr)] | ||
#[derive(Deserialize, Serialize)] | ||
pub enum SplitState { | ||
Idle = 0, | ||
Splitting = 1, | ||
} | ||
|
||
impl Default for SplitState { | ||
fn default() -> Self { | ||
Self::Idle | ||
} | ||
} | ||
|
||
type SplitStateSQLRepr = Int2; | ||
|
||
impl ToSql<SplitStateSQLRepr, Pg> for SplitState { | ||
fn to_sql<'a>( | ||
&'a self, | ||
out: &'a mut diesel::serialize::Output<Pg>, | ||
) -> diesel::serialize::Result { | ||
let raw_value: i16 = *self as i16; | ||
let mut new_out = out.reborrow(); | ||
ToSql::<SplitStateSQLRepr, Pg>::to_sql(&raw_value, &mut new_out) | ||
} | ||
} | ||
|
||
impl FromSql<SplitStateSQLRepr, Pg> for SplitState { | ||
fn from_sql(pg_value: PgValue) -> diesel::deserialize::Result<Self> { | ||
match FromSql::<SplitStateSQLRepr, Pg>::from_sql(pg_value).map(|v| match v { | ||
0 => Some(Self::Idle), | ||
1 => Some(Self::Splitting), | ||
_ => None, | ||
})? { | ||
Some(v) => Ok(v), | ||
None => Err(format!("Invalid SplitState value, was: {:?}", pg_value.as_bytes()).into()), | ||
} | ||
} | ||
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
af91a28
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
2475 tests run: 2351 passed, 0 failed, 124 skipped (full report)
Code coverage (full report)
functions
:54.3% (11471 of 21106 functions)
lines
:81.6% (64431 of 78972 lines)
af91a28 at 2024-02-08T17:12:00.826Z :recycle: