This pipeline demonstrates how to bulk load data from MySQL into Azure Synapse SQL. You can select one or more MySQL schemas to replicate into Synapse SQL with a single Data Collector pipeline.
For more information, see Loading Data into Azure Synapse SQL in StreamSets Data Collector documentation.
- Run StreamSets Data Collector from Azure Marketplace or download it for local use. Download version must be 3.19.0 or higher.
- Ensure the pre-requisites for Azure Synapse SQL are complete
- Download the pipeline and import it into Data Collector or Control Hub
- Configure all the pipeline parameters for your MySQL Database, Synapse and Azure Staging connections.
- Optionally, update the MySQL origin to read from specific schemas and/or tables
- By default, the Azure Synapse SQL destination is configured to auto create each table that is replicated from MySQL. If you'd like, update the configurations in the destination per your needs.
Start the pipeline. It takes a couple of seconds to create a connection to Synapse. Once the connection is established, you should see records ingested from MySQL and sent to Synapse SQL. The pipeline is also configured to automatically stop after it's ingested all the data from MySQL. If you'd like to continue running the pipeline such that it incrementally ingests as data arrives in MySQL then uncheck 'Process Events' from the origin and delete the 'Load complete' executor.
