Code

Take Home Exam Solution#1

Code

Open

1byte-yoda wants to merge 121 commits intoZookal:masterZookal/dataengineer_test:masterfrom 1byte-yoda:developCopy head branch name to clipboard

+2,697-200

Commits

Commits on Jul 28, 2021

• Added idea folder in gitignore

  

  Mark

  committed
  5fdfdde

  Copy full SHA for 5fdfdde

• Deleted data.zip to save storage

  

  Mark

  committed
  0241e85

  Copy full SHA for 0241e85

• Development Checklist for Continuous Integration

  

  Mark

  committed
  4d17309

  Copy full SHA for 4d17309

• Folder blueprint for development

  

  Mark

  committed
  f0f41d9

  Copy full SHA for f0f41d9

• Docker compose setup

  

  Mark

  committed
  2734723

  Copy full SHA for 2734723

• Added Project Package Requirements

  

  Mark

  committed
  51b2978

  Copy full SHA for 51b2978

• Git ignore logs folder

  

  Mark

  committed
  df53a47

  Copy full SHA for df53a47

• Renamed README.md to instructions.md

  

  Mark

  committed
  b8b5eb8

  Copy full SHA for b8b5eb8

• Renamed README.md to instructions.md

  

  Mark

  committed
  a5bcb3c

  Copy full SHA for a5bcb3c

• Added the project's manual

  

  Mark

  committed
  8f090a1

  Copy full SHA for 8f090a1

• Renamed README.md to test-instructions.md

  

  Mark

  committed
  1785d63

  Copy full SHA for 1785d63

• Flake8 and Pytest configuration

  

  Mark

  committed
  b1f9065

  Copy full SHA for b1f9065

• Spark package where Spark Jobs will reside, and deleted airflow config files.

  

  Mark

  committed
  ae24fb0

  Copy full SHA for ae24fb0

• Added Note for docker-compose Postgresql execution, airflow web server must wait postgres before it runs

  

  Mark

  committed
  2190b52

  Copy full SHA for 2190b52

• Adde pyspark in the project's package requirements

  

  Mark

  committed
  5e5d8b9

  Copy full SHA for 5e5d8b9

• Docker config for Spark with 3 executors

  

  Mark

  committed
  0d57403

  Copy full SHA for 0d57403

• Added a note for future development

  

  Mark

  committed
  f8c9607

  Copy full SHA for f8c9607

• Git Ignore Pycharm Config

  

  Mark

  committed
  25170f5

  Copy full SHA for 25170f5

• Added TODO notes in README

  

  Mark

  committed
  c0015ed

  Copy full SHA for c0015ed

Commits on Jul 29, 2021

• Fixed airflow scheduler is failing when database creation is not yet finished

  

  Mark

  committed
  33507d8

  Copy full SHA for 33507d8

• Added Development Notes

  

  Mark

  committed
  fdeea40

  Copy full SHA for fdeea40

• Upgraded airflow version, and reverted back pyspark to 3.0.1

  

  Mark

  committed
  331778f

  Copy full SHA for 331778f

• Retail dag auto formatted by black

  

  Mark

  committed
  9cbaa96

  Copy full SHA for 9cbaa96

• Docker configuration for Spark added in airflow Dockerfile, since we are running in client mode and our client is airflow's machine

  

  Mark

  committed
  5740f82

  Copy full SHA for 5740f82

• Added example dag for spark job

  

  Mark

  committed
  7724086

  Copy full SHA for 7724086

• Added Kafka in the Docker registry

  

  Mark

  committed
  0e48e1f

  Copy full SHA for 0e48e1f

• Added log4j config for Kafka Logs

  

  Mark

  committed
  e35cb40

  Copy full SHA for e35cb40

• Added Kafka Python library in the package requirements

  

  Mark

  committed
  2f1828a

  Copy full SHA for 2f1828a

• Added data folder in git ignore file

  

  Mark

  committed
  0247765

  Copy full SHA for 0247765

• Added few more development notes for the future

  

  Mark

  committed
  948919c

  Copy full SHA for 948919c

• Added mysqlclient and mysqlconnector to support airflow's mysql hook installation

  

  Mark

  committed
  3f84053

  Copy full SHA for 3f84053

• Data generation (?)

  

  Mark

  committed
  441f848

  Copy full SHA for 441f848

• Initialized data sync operator to test the MySQLHook installed

  

  Mark

  committed
  a49e46c

  Copy full SHA for a49e46c

• Separate Dockerfile for MySQL service.

  

  Mark

  committed
  d049b17

  Copy full SHA for d049b17

• Black Auto Formatting

  

  Mark

  committed
  e6e9f6e

  Copy full SHA for e6e9f6e

• Added development env file

  

  Mark

  committed
  a62df65

  Copy full SHA for a62df65

• Fixed Checklist Lints

  

  Mark

  committed
  6cc637b

  Copy full SHA for 6cc637b

• Data Generation (???)

  

  Mark

  committed
  82036b2

  Copy full SHA for 82036b2

Commits on Jul 30, 2021

• Added notes for documentation later on

  

  Mark

  committed
  a5f12db

  Copy full SHA for a5f12db

• Added clean up of cache objects in make cmd

  

  Mark

  committed
  505d612

  Copy full SHA for 505d612

• Remove synchronization of data folder to docker container

  

  Mark

  committed
  749185f

  Copy full SHA for 749185f

• Tbl to Stage Operator setup

  

  Mark

  committed
  e585df7

  Copy full SHA for e585df7

• Added pandas in package requirements

  

  Mark

  committed
  0f91bd2

  Copy full SHA for 0f91bd2

• Renamed ORDERS table to ORDER

  

  Mark

  committed
  4e02172

  Copy full SHA for 4e02172

• TblToStage Operator implementation

  Show description for 9ab60f8
  

  Mark

  committed
  9ab60f8

  Copy full SHA for 9ab60f8

• Black Autoformatting

  

  Mark

  committed
  b739847

  Copy full SHA for b739847

• Added Upsert Query for region table

  

  Mark

  committed
  ec6bb8e

  Copy full SHA for ec6bb8e

• Renamed table names into lowercase

  

  Mark

  committed
  880841c

  Copy full SHA for 880841c

• Renamed SQL table names into lowercase

  

  Mark

  committed
  628a9c8

  Copy full SHA for 628a9c8

• Removed Kafka and Upgraded postgres into v13

  

  Mark

  committed
  1ca3572

  Copy full SHA for 1ca3572

Commits on Jul 31, 2021

• Configured automated security check and static type checking

  

  Mark

  committed
  35aa39d

  Copy full SHA for 35aa39d

• Added a command for pruning docker containers

  

  Mark

  committed
  4358d2e

  Copy full SHA for 4358d2e

• Replaced the usage of pandas to_sql for inserting.

  Show description for 69448e0
  

  Mark

  committed
  69448e0

  Copy full SHA for 69448e0

• Modularization of TblToStageOperator class

  Show description for a498131
  

  Mark

  committed
  a498131

  Copy full SHA for a498131

• Correction on each of items in the checklist configuration

  

  Mark

  committed
  f2e98e0

  Copy full SHA for f2e98e0

• Pytest plugin for dockerized testing

  

  Mark

  committed
  c416dcc

  Copy full SHA for c416dcc

• Removed variables in conftest.py

  

  Mark

  committed
  6b24f1a

  Copy full SHA for 6b24f1a

• Added unit tests for general rules for each DAGs

  

  Mark

  committed
  f2bc149

  Copy full SHA for f2bc149

• Added unit tests for the retail_dag DAG

  

  Mark

  committed
  7f5bd13

  Copy full SHA for 7f5bd13

• Removed unused variables in dag integrity test suite

  

  Mark

  committed
  9ae3bc2

  Copy full SHA for 9ae3bc2

• Black Autoformatting

  

  Mark

  committed
  3ec82ec

  Copy full SHA for 3ec82ec

• Sample dag for parallel spark job tasks

  

  Mark

  committed
  59ca313

  Copy full SHA for 59ca313

• New Integration Tests for TblToStageOperator

  

  Mark

  committed
  688b2c0

  Copy full SHA for 688b2c0

• Added pytest-docker in package requirements for dockerized testing

  

  Mark

  committed
  6388788

  Copy full SHA for 6388788

• Added documentation for common errors

  

  Mark

  committed
  1009b23

  Copy full SHA for 1009b23

• Enhanced the static typing for TblToStageOperator.execute() method

  

  Mark

  committed
  b82ff26

  Copy full SHA for b82ff26

• Removed Kafka Package Dependency

  

  Mark

  committed
  4649e78

  Copy full SHA for 4649e78

• Added more notes for documentation purposes.

  

  Mark

  committed
  987057b

  Copy full SHA for 987057b

• Adjustment on test for dag integrity due to changes in dag definition

  

  Mark

  committed
  7db71f7

  Copy full SHA for 7db71f7

• Added a getter function to get upsert queries for a given table name.

  

  Mark

  committed
  8218ad5

  Copy full SHA for 8218ad5

• Clean up retail_dag.py

  

  Mark

  committed
  6f78f01

  Copy full SHA for 6f78f01

• Wrote an instruction to mitigate issues when running tests

  

  Mark

  committed
  bd81a43

  Copy full SHA for bd81a43

• Added a test for idempotency, also DRYed the code for cleaner tests.

  

  Mark

  committed
  4d03d41

  Copy full SHA for 4d03d41

Commits on Aug 1, 2021

• Added INSERT queries for data insertion to MySQL

  

  Mark

  committed
  dfa255a

  Copy full SHA for dfa255a

• Added execution time for query table filtering

  Show description for 44ea6a2
  

  Mark

  committed
  44ea6a2

  Copy full SHA for 44ea6a2

• Modification of typo errors in SupplierHeader model

  

  Mark

  committed
  18fef67

  Copy full SHA for 18fef67

• Black Autoformatting

  

  Mark

  committed
  cc74896

  Copy full SHA for cc74896

• Improved the ddl operations by adding if not exists upon creation.

  

  Mark

  committed
  bb9a2fc

  Copy full SHA for bb9a2fc

• Restructured DAG connections

  

  Mark

  committed
  c9c532a

  Copy full SHA for c9c532a

• Black Autoformatting

  

  Mark

  committed
  f5623bc

  Copy full SHA for f5623bc

• Fixed test errors for dag test suites.

  

  Mark

  committed
  ba99845

  Copy full SHA for ba99845

• Reduced verbosity of pytest

  

  Mark

  committed
  7c18b63

  Copy full SHA for 7c18b63

• Fixed unit test errors for TblToStageOperator

  

  Mark

  committed
  3818765

  Copy full SHA for 3818765

• Merge pull request #1 from 1byte-yoda/mark/data-ingester-pipeline

  Show description for 38baba6
  1byte-yoda

  authored
  38baba6

  Copy full SHA for 38baba6

• Merge pull request #1 from 1byte-yoda/mark/data-ingester-pipeline

  Show description for 5107390
  

  1byte-yoda

  authored and

  Mark

  committed
  5107390

  Copy full SHA for 5107390

• Modified the commit message for the merged branch.

  

  Mark

  committed
  493e072

  Copy full SHA for 493e072

• Removed PySpark from the project

  

  Mark

  committed
  9dde073

  Copy full SHA for 9dde073

• Removed spark volume binding in docker

  

  Mark

  committed
  12c4229

  Copy full SHA for 12c4229

• Added PostgreSQL requirements for development

  Show description for f6096fd
  

  Mark

  committed
  f6096fd

  Copy full SHA for f6096fd

• Health check for Postgres DW

  

  Mark

  committed
  810700b

  Copy full SHA for 810700b

• Optional return for get_upsert_query method

  

  Mark

  committed
  70304ca

  Copy full SHA for 70304ca

Commits on Aug 2, 2021

• Removed unused files / irrelevant to project

  

  Mark

  committed
  bd819f4

  Copy full SHA for bd819f4

• Added a Step-by-Step guide to run the project.

  Show description for 6913d84
  

  Mark

  committed
  6913d84

  Copy full SHA for 6913d84

• Postgres to Datawarehouse Operator Implementation

  Show description for e0b8042
  

  Mark

  committed
  e0b8042

  Copy full SHA for e0b8042

• Clean up and added some TODO notes for later development use.

  

  Mark

  committed
  5723147

  Copy full SHA for 5723147

• Added SQL Analytical Queries that will serve as an anwer for the Reporting questions.

  

  Mark

  committed
  d46f64e

  Copy full SHA for d46f64e

• Initial test setup for postgres_dw_operator

  

  Mark

  committed
  dc3fdb4

  Copy full SHA for dc3fdb4

• Decoupled environment variables for docker-compose services

  

  Mark

  committed
  b30c878

  Copy full SHA for b30c878

• README minimal typo error

  

  Mark

  committed
  c46a3c8

  Copy full SHA for c46a3c8

• Removed extra/unuseful texts in README.md

  

  Mark

  committed
  b45cef1

  Copy full SHA for b45cef1

• Cleanup TODO lists

  

  Mark

  committed
  772d2c4

  Copy full SHA for 772d2c4

• Added Answers for each exam questions

  

  Mark

  committed
  aab3bfd

  Copy full SHA for aab3bfd

• Removed some unnecessary details in README and ANSWERS file

  

  Mark

  committed
  d1eaf54

  Copy full SHA for d1eaf54

• Removed docker environment variables in git ignored files

  

  Mark

  committed
  1b1b26f

  Copy full SHA for 1b1b26f

• Added a link in the ANSWERS.md file for SQL Queries for Dashboard Reports.

  

  Mark

  committed
  cd554cb

  Copy full SHA for cd554cb

• Format SQL query for better readability

  

  Mark

  committed
  464be36

  Copy full SHA for 464be36

• Fixed formatting issues in ANSWERS.md

  

  Mark

  committed
  f20024c

  Copy full SHA for f20024c

• Fixed some line indentions

  

  Mark

  committed
  423aca2

  Copy full SHA for 423aca2

• Fixed Checklist Errors

  

  Mark

  committed
  e146119

  Copy full SHA for e146119

• Fixed typo error in dashboard queries

  

  Mark

  committed
  86c1b5e

  Copy full SHA for 86c1b5e

• Fixed some typo issue in ANSWERS.md file

  

  Mark

  committed
  5223044

  Copy full SHA for 5223044

• Added more make commands in README.md file

  

  Mark

  committed
  1ba1cbb

  Copy full SHA for 1ba1cbb

• Improved SQL Query variable naming

  

  Mark

  committed
  30dcb30

  Copy full SHA for 30dcb30

• Improvements in the Table DDL file formatting

  

  Mark

  committed
  ac36a35

  Copy full SHA for ac36a35

Commits on Aug 4, 2021

• Added surrogate key for each dimension tables and also adjusted the fact table.

  

  Mark

  committed
  1634b49

  Copy full SHA for 1634b49

• Fixed flake8 lints

  

  Mark

  committed
  cb6eff7

  Copy full SHA for cb6eff7

• Fixed Checklist lints

  

  Mark

  committed
  6ec0182

  Copy full SHA for 6ec0182

• Improved star schema by adding data types

  

  Mark

  committed
  1388bde

  Copy full SHA for 1388bde

• Added key notations for start schema for better readability

  

  Mark

  committed
  492e8f0

  Copy full SHA for 492e8f0

• Optimized dag batch run to 15,000 rows per chunks

  

  Mark

  committed
  eefe524

  Copy full SHA for eefe524

• Fixed checklist errors and optimized sql queries for auto incremented id insertion

  

  Mark

  committed
  6d098da

  Copy full SHA for 6d098da