Sales-DataPipeline

---

Project Overview

This project is an overview of an Sales Data Projection data pipeline that involves near real tine data ingestion and transformation with Change Data Capture functionality. We will design a system using AWS services such as S3, Lambda, Glue, DynamoDB, Kinesis Stream, Kinesis Firehose and Event Bridge to ingest, transform, with change data capture functionality to load data in S3 and accessing using Athena for analytical purposes.

---

Architectural Diagram

---

Key Steps

1. Create a Table "OrdersRawTable" in DynamoDB

2. Create a Data Stream "kinesis-for-sales-data" in Kinesis.

3. Create a Event Bridge Pipe to ingest data from DynamoDb Stream to Kinesis Stream.

• Note: Give the attached IAM role the permission to access DynamoDB & Kinesis
  • AmazonDynamoDBFullAccess
  • AmazonKinesisFullAccess

4. Run the Mock data generator script to load the data in DyanomoDB.

5. Check the Data Load

• Data should be visible in DynamoDB Table "OrdersRawTable"

• Event Bridge should be triggerd and data should flow from DyanomoDB Stream to Kinesis Stream

6. Check the Change Data Capture (CDC) Events

• Before Edit the data

  • 
  • 

• After Edit the data

  • 
  • 
  • 

7. Create Kinesis Firehose

• Create Kinesis Firehose to fetch the data from Kinesis Stream and transform it with the help of Lambda and load as batches into S3

• Kinesis Firehose

  • 

• Lambda Function for Transformation

  • 

• S3 Bucket "kinesis-firehose-destination-yb" for data load destination

  • 

8. Generate Some more mock data in DynamoDB

• The data will be generated by mock data generator script and data will flow from DynamoDB to Kinesis Stream. from the Kinesis stream the daat will flow in Kinesis Firehose and Tranform by Lambda function and stored in the destination S3 bucket
  • 

9. Create a Glue Crawler to crawl data from destination S3 bucket

• 

  • Note: Create a Classifier and attach to crawler to load the JSON data in the correct format
  • 

• Schema fetched by Crawler

  • 

10. Query the data from S3 using Athena and Glue's Crawled Schema