Implement Destination #30
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
hariso
reviewed
Feb 13, 2023
oykmnk
force-pushed
the
source
branch
6 times, most recently
from
91c1205 to
3530c9d
Compare
hariso
reviewed
Mar 1, 2023
hariso
reviewed
Mar 1, 2023
hariso
approved these changes
Mar 1, 2023
voscob
added a commit
that referenced
this pull request
Mar 2, 2023
* Implement the common config and validator (#1) * Implement the Source config (#2) * Implement a Source base (#3) * Implement the CDC iterator (#5) This patch seeks to implement the CDC iterator. It works by creating and listening to a MongoDB Change Stream. The Change Stream can be resumed by a `resumeToken` that every Change Stream event has in its payload. The CDC iterator stores the `resumeToken` in the Position. * Implement CDC integration tests (#6) This patch seeks to implement integration tests for CDC. * Update the `Config.URI` field (#9) * Implement the Snapshot iterator (#10) * Rename `snapshotMode` to `snapshot` (#11) * Update snapshot iterator, rename `orderingColumn` to `orderingField` (#13) * Add client options and custom BSON codec to Source (#15) * Add Source documentation (#16) * Add database and collection validations (#18) * store resumeToken in a snapshot position (#19) * Update Mongo client options (#22) * Update `GetMongoCollection` function (#23) * Implement polling snapshot functionality (#26) This patch seeks to implement polling snapshot functionality. It's used when the full CDC is not available. For instance, Azure CosmosDB for MongoDB has very limited support for Change Streams, so they cannot be used within the connector. Signed-off-by: Aleksandr Yakimenko <[email protected]> * register slice converter * Remove redundant dot (#34) Signed-off-by: Aleksandr Yakimenko <[email protected]> * Fix polling snapshot initialization (#35) * fixes after review Signed-off-by: Aleksandr Yakimenko <[email protected]> * Update .gitignore * Update Source Parameters with sdk.Validation * Add the comment for the initialisation of Snapshot iterator * Add test for EncodeValue of StringObjectIDCodec * Add t.Parallel to the test for EncodeValue of StringObjectIDCodec * Update AuthMechanism Source Parameters * Update modules * Update test of EncodeValue method * Add tests for EncodeValue method * Implement Destination (#30) * Implement a Destination base * Implement the Writer for the Destination (#8) * Implement acceptance test (#14) * Add Destination documentation (#17) * Destination integration test (#12) * Add error check to the destination_test * Rename the Write return variable * Add test-integration command * Update Destination Parameters with sdk.Validation * Add a few data types to acceptance test * Update AuthMechanism Destination Parameters * Update AuthMechanism Destination Parameters * Add Integration test step to the build github action * Remove test constant from command package --------- Co-authored-by: BohdanMyronchuk <[email protected]> Co-authored-by: Yurii Voskoboinikov <[email protected]> --------- Signed-off-by: Aleksandr Yakimenko <[email protected]> Co-authored-by: Yurii Voskoboinikov <[email protected]> Co-authored-by: BohdanMyronchuk <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This PR seeks to implement Destination logic.
Quick checks: