Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature]: Support MySQL full-database synchronization or multi-table synchronization. #22

Open
2 tasks
hantmac opened this issue Nov 29, 2023 · 1 comment · May be fixed by #27
Open
2 tasks

[Feature]: Support MySQL full-database synchronization or multi-table synchronization. #22

hantmac opened this issue Nov 29, 2023 · 1 comment · May be fixed by #27
Assignees

Comments

@hantmac
Copy link
Collaborator

hantmac commented Nov 29, 2023

  • Support full-database synchronization.

  • The table-name also need to support regular expressions to monitor multiple tables that satisfy the regular expressions.

@hantmac hantmac self-assigned this Nov 29, 2023
@hantmac hantmac changed the title [Feature]: Support full-database synchronization or multi-table synchronization. [Feature]: Support MySQL full-database synchronization or multi-table synchronization. Jan 30, 2024
@hantmac
Copy link
Collaborator Author

hantmac commented Jan 30, 2024

This can based on the Flink DataStream package, which can implement a Flink task for submitting a whole database/multi-table synchronization by executing the command line.
The main process is as follows:

  1. Obtain the MySQL link information, and read the table that needs to be synchronized this time according to the configuration
  2. Determine whether the table exists in Databend, if not, create the table
  3. Obtain CDC upstream data, and use the side output stream for distribution according to the table name
  4. Create a DatabendSink for each stream.
  5. Submit FlinkJob

@hantmac hantmac linked a pull request Jan 30, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant