AWS-Orchestrating the Data Pipeline-Chapter 10

Description

This Chapter will help us to understand :

1. How to decide the best option for Data Pipeline orchestation on AWS between MWAA(Airflow) & AWS Step Functions ?
2. DEMO - 1
3. Study case about - How to extract data from Youtube Channel and orchestrate it with AWS Step functions
4. Data Stats Channel
5. Data Video Features

How to decide the best option for Data Pipeline orchestation ?

DEMO - 1

For our Step Function state machine, we are going to start by using a Lambda function that checks the extension of an incoming file to determine the type of file. Once determined, we'll pass that information on to the next state, which is a CHOICE state. If it is a file type we support, we'll call a Lambda function to process the file, but if it's not, we'll send out a notification, indicating that we cannot process the file.

If the Lambda function fails, we'll send a notification to report on the failure; otherwise, we will end the state machine with a SUCCESS status.

How to extract data from Youtube Channel and orchestrate it with AWS Step functions ?

Data Stats Channel

• Go to Data 👉 Stats Channel

Data Video Features

• Go to Data 👉 Video Featuresl