Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix kafka avro wire format #2356

Merged
merged 9 commits into from
Oct 16, 2023
Merged

Fix kafka avro wire format #2356

merged 9 commits into from
Oct 16, 2023

Conversation

Licenser
Copy link
Member

@Licenser Licenser commented Oct 12, 2023

Pull request

Description

  • remove schema registry preprocessor
  • remove index lookup and schema registry lookup in avro codec
  • add confluent_schema_registry codec for avro
  • move pre and post-processors to tremor-interceptor
  • add tremor-config

Checklist

  • The RFC, if required, has been submitted and approved
  • Any user-facing impact of the changes is reflected in docs.tremor.rs
  • The code is tested
  • Use of unsafe code is reasoned about in a comment
  • Update CHANGELOG.md appropriately, recording any changes, bug fixes, or other observable changes in behavior
  • The performance impact of the change is measured (see below)

Performance

Signed-off-by: Heinz N. Gies <[email protected]>
Signed-off-by: Heinz N. Gies <[email protected]>
Signed-off-by: Heinz N. Gies <[email protected]>
Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser Licenser force-pushed the fix-kafka-avro-wire-format branch 6 times, most recently from 0205203 to 6c4d872 Compare October 13, 2023 12:33
@codecov
Copy link

codecov bot commented Oct 13, 2023

Codecov Report

Merging #2356 (2d1aa1a) into main (92b1f0d) will increase coverage by 0.01%.
The diff coverage is 87.46%.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #2356      +/-   ##
==========================================
+ Coverage   90.87%   90.88%   +0.01%     
==========================================
  Files         279      281       +2     
  Lines       59553    59675     +122     
==========================================
+ Hits        54117    54238     +121     
- Misses       5436     5437       +1     
Flag Coverage Δ
unittests 90.88% <87.46%> (+0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Coverage Δ
src/config.rs 98.35% <100.00%> (+0.04%) ⬆️
src/connectors/google.rs 88.70% <100.00%> (ø)
src/connectors/impls/bench.rs 86.78% <100.00%> (ø)
src/connectors/impls/cb.rs 90.70% <100.00%> (+0.16%) ⬆️
src/connectors/impls/clickhouse.rs 97.17% <100.00%> (ø)
src/connectors/impls/crononome.rs 95.60% <100.00%> (ø)
src/connectors/impls/dns/client.rs 82.56% <100.00%> (ø)
src/connectors/impls/elastic.rs 94.28% <100.00%> (ø)
src/connectors/impls/exit.rs 97.40% <100.00%> (ø)
src/connectors/impls/file.rs 91.79% <100.00%> (ø)
... and 99 more

... and 2 files with indirect coverage changes


Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 92b1f0d...2d1aa1a. Read the comment docs.

@Licenser Licenser force-pushed the fix-kafka-avro-wire-format branch 3 times, most recently from b13ed01 to 9f55021 Compare October 13, 2023 13:32
Signed-off-by: Heinz N. Gies <[email protected]>
mfelsche
mfelsche previously approved these changes Oct 14, 2023
Copy link
Member

@mfelsche mfelsche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some minor typoes, otherwise 👍

tremor-codec/src/codec/avro.rs Outdated Show resolved Hide resolved
tremor-codec/src/codec/avro.rs Outdated Show resolved Hide resolved
tremor-codec/src/codec/kafka_schema_registry.rs Outdated Show resolved Hide resolved
tremor-codec/src/codec/kafka_schema_registry.rs Outdated Show resolved Hide resolved
tremor-codec/src/codec/kafka_schema_registry.rs Outdated Show resolved Hide resolved
tremor-codec/src/codec/avro.rs Outdated Show resolved Hide resolved
tremor-codec/src/lib.rs Outdated Show resolved Hide resolved
tremor-interceptor/Cargo.toml Outdated Show resolved Hide resolved
Co-authored-by: Matthias Wahl <[email protected]>
Signed-off-by: Heinz N. Gies <[email protected]>
Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser
Copy link
Member Author

Licenser commented Oct 16, 2023

All fixed :) thanks for the detailed review!

Signed-off-by: Heinz N. Gies <[email protected]>
@Licenser Licenser merged commit 81897ea into main Oct 16, 2023
18 checks passed
@Licenser Licenser deleted the fix-kafka-avro-wire-format branch October 16, 2023 09:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants