Replace TenantIndex, BlockMeta and CompactedBlockMeta with proto near-equivalents #4072
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zalegrala
force-pushed
the
jsonpbTenantIndex
branch
3 times, most recently
from
513e33b to
3c10c0f
Compare
zalegrala
changed the title
zalegrala
force-pushed
the
jsonpbTenantIndex
branch
2 times, most recently
from
f65adc7 to
cc764f5
Compare
joe-elliott
reviewed
Sep 19, 2024
zalegrala
force-pushed
the
jsonpbTenantIndex
branch
4 times, most recently
from
5d44486 to
c89cb07
Compare
zalegrala
requested review from
annanay25,
mdisibio,
mapno,
yvrhdn,
electron0zero,
ie-pham and
stoewer
as code owners
mdisibio
reviewed
Sep 30, 2024
There was a problem hiding this comment.
I like where we ended up with this PR. The existing block meta.json and block state are unchanged, but we will get the index proto efficiency on the downstream components (frontend, querier). This PR looks like a lot, but 90% is fallout from the uuid and Size changes. The actual change is much smaller. 99% LGTM left a few q's and comments.
mdisibio
approved these changes
Oct 2, 2024
zalegrala
force-pushed
the
jsonpbTenantIndex
branch
from
4cdb387 to
89c0a71
Compare
3 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does:
Here we replace the
TenantIndex,BlockMetaandCompactedBlockMetawith structs which are generated from proto. This allows us to convert theTenantIndexto proto-only without the translation between a backend struct and an internal struct, since the internal structs here are being replaced. TheBlockMetaandCompactedBlockMetaremainjson. We may consider replacing these with proto in the future.The replacement structs are not identical. Of note, the
BlockMetastruct has several fields which have no proto equivalent. In order to overcome this, several types have been defined in the proto asbytes, which allows the use of thecustomtypegogo proto option to point to a specific type. Those types must conform to a specific interface, and so methods have been added where necessary.backend.DedicatedColumnsupdated to include proto marshal and unmarshal methodsbackend.Encodingupdated to include prto marshal and unmarshal methodsbackend.BlockMetahas a few type changesTotalObjectschangesint->int64Sizebecomes a funcion and the old size is moved toSize_CompactionLevelchangesuint8->uint32BloomShardCountchangesuint16->uint32ReplicationFactorchangesuint8->uint32A new UUID type has been added which is backed by the google UUID type and provide
the necessary interface for use with proto.
The
TenantIndexis written both as proto and json, and read as protofirst with a fallback to json on error.The
BlockMetaandCompactedBlockMetaare written/read only as json.Which issue(s) this PR fixes:
Fixes #
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX]