Features

• pep8 compliant
• scheduled ETL processes
• integration with FHIR-compatible databases/APIs
• data validation
• logging for better monitoring and observability
• error handling
• parallelization-capable
• incremental-loading (WIP)

---

Instructions

Environment

• make sure you have docker installed

• make sure you have docker-compose installed

• make sure you have python 3.9 installed

• make sure you have the requirements.txt installed

• Create a .env file in the root directory with the following content:

• POSTGRES_HOST: the host of the PostgreSQL database

• POSTGRES_PORT: the port of the PostgreSQL database

• POSTGRES_DB: the name of the PostgreSQL database

• POSTGRES_USER: the username of the PostgreSQL database

• POSTGRES_PASSWORD: the password of the PostgreSQL database

example:

POSTGRES_HOST={your-host}
POSTGRES_DB={your-db}
POSTGRES_USER={your-user}
POSTGRES_PASSWORD={your-password}
POSTGRES_PORT={your-port}

data prep

• unzip data/patients_fhir_100.zip

Docker

• run docker build -t mendel/app . to build the app image
• run docker-compose up --build to build and run the container
• that's it! you're all set.
• you can use psql -h localhost -d mendel -U mendel -p 5432 in the database container terminal to connect to the database
• \dt
• select * from information_schema.tables;
• ...

Local

• pip install -r requirements.txt
• run python scheduler.py to run the scheduler, you can select the interval and time from main.py
• run python main.py to run the whole project with the scheduled ETL processes
• run python processor.py to run the data extraction and transformation logic and update the CSV files in out/
• run python loader.py to run the data loading logic and upload the CSV files to the database

---

Database Schema

---

Technologies

• Synthea
• FHIR
• json crack
• Pandas
• SQLAlchemy
• Requests
• Pytest
• Docker
• scheduler