Introduce MaterializationEndpoint::Dekaf
#1642
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jshearer
changed the title
MaterializationEndpoint::Dekaf
jshearer
force-pushed
the
jshearer/dekaf_materialization_endpoint
branch
2 times, most recently
from
96653f0
to
000a3a0
Compare
jshearer
force-pushed
the
jshearer/dekaf_materialization_endpoint
branch
from
cf898b0
to
6fa871d
Compare
jgraettinger
approved these changes
Sep 23, 2024
jshearer
force-pushed
the
jshearer/dekaf_materialization_endpoint
branch
from
78cae66
to
5a1f419
Compare
jshearer
commented
Sep 24, 2024
| @@ -41,6 +41,13 @@ pub struct MaterializationDef { | |||
| pub delete: bool, | |||
| } | |||
|
|
|||
| #[derive(Serialize, Deserialize, Clone, Debug, JsonSchema, PartialEq)] | |||
| #[serde(untagged)] | |||
| pub enum DekafConfigContainer { | |||
There was a problem hiding this comment.
@jgraettinger This seem right to you? The other endpoint types use a RawValue for their config because they're dynamic, and that lets them inherently accept either a string (which is treated as an indirect reference), or an object (which is treated as an inline config), whereas the dekaf config is defined in-tree.
I contemplated whether I should generalize this Direct / Indirect config container to the other endpoint types to make that distinction more explicit, but it didn't seem warranted, more of a preference than tech debt. Thoughts?
jshearer
force-pushed
the
jshearer/dekaf_materialization_endpoint
branch
2 times, most recently
from
b028eb4
to
47423d0
Compare
A materialization of this kind is purely descriptive, and cannot be directly started. It is used to convey the intent to expose a set of bindings through Dekaf, which will look it up to determine things like binding names and field selection/projections.
…very journals, but they _should_ get ops stats+logs collections
jshearer
force-pushed
the
jshearer/dekaf_materialization_endpoint
branch
from
47423d0
to
1a20b83
Compare
This was referenced Sep 26, 2024
|
Closing in favor of #1665 which contains some substantial changes from this PR which I didn't want to push here and confuse things |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
This is the first part of #1622. It introduces a new materialization endpoint type called
dekaf, as well as the necessary control-plane plumbing to give materializations of this type ops logs and stats journals when activated. As materializations of this type are purely descriptive and are actually served by the stand-alone Dekaf service, no shards or recovery log journals should be created.I added a new happy-path test to snapshot a new dekaf-type materialization, and am able to publish the following spec to my local stack, see it show up in the UI, and delete it successfully.
This change is