Automated Ingestion Script

General Info

This script can ingest .csv data into postgresql databases . The aim of the project however, is to ingest data of any format into any database. At present, working on the plugins for different formats (i.e. inputs) and databases (i.e. outputs).

Technologies

The script is developed in Python.

Setup

1. Modify src/database.ini with your own database credentials.
2. Drop the data into data/to_ingest folder.
3. Add schema to the command variable inside create_table.py file.
4. Run ingest.sh

and DONE!! The data is ingested.

Status

1. Working to create plugins that allow ingesting data of any type (such as .parquet, .xml, .xlsx, .tsv).
2. Also working to create plugins that allow ingesting into any type of database (such as PostgreSQL, mySQL, MariaSQL, MSSQL)