Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Schema Processor Revamp implementation #35213

Closed

Conversation

ankitpatel96
Copy link
Contributor

@ankitpatel96 ankitpatel96 commented Sep 16, 2024

Description:

This is the schema processor, as originally written by @MovieStoreGuy in #11547 - refreshed and rebased. I've rewritten some parts and added significant amounts of testing to many parts - resulting in quite a few changes. I will be splitting this PR into many parts.

There are several features that need to be completed before this is ready for production use:

  1. This only supports the Schema File Format 1.0.0 - we need to support 1.1.0
  2. Right now only downgrades work. Upgrades need some work to enable
  3. We should improve internal metrics and observability.

MovieStoreGuy and others added 30 commits July 31, 2024 16:11
As the schema processor improves, this package path changes to help make
sense of what is trying to do.
In order to help validate the works of the schema transformer in a
future PR, these additional packages add in functionality to verify /
abstract efforts.
A modifier allows mutating a signal from a previous version to the
current one and vice versa.
A revision handles applying the modifiers to incoming signals and
converting a signal to the next version.
A translation is an immutable representation of the schema translation
file that can be applied to each imcoming signal that is then managed by
the schema manager.

The schema manager has a set of providers that allow the schema
translation to be fetched from the remote location.
@github-actions github-actions bot added the processor/schema Schema processor label Sep 16, 2024
@ankitpatel96 ankitpatel96 changed the title Schema Processor implementation Schema Processor Revamp implementation Sep 16, 2024
@ankitpatel96 ankitpatel96 marked this pull request as ready for review September 16, 2024 22:45
@ankitpatel96 ankitpatel96 requested a review from a team September 16, 2024 22:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
processor/schema Schema processor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants