Demeter: An Automatic Framework for Data Migration in Open Data Lakes

Components

• Open-data manager
• Airbyte-manager

---

An Open Data Lake (ODL) stores various forms and types of open data, and there is an increasing demand to manage raw data in tables rather than files for efficient data exploration and analysis. We investigate the data management of ODL and recognize the limitations of table migration and related problems. ODL has three problems below.

1. pre-processing complexity
2. scale limitation
3. platform dependency

Existing studies for table migration have three problems below.

1. lack of scalability
2. migration incompleteness
3. scale limitation

We present a novel automation framework, called Demeter, which solves three problems inherent in ODL by expanding automation.