🚚 AP data discovery and classification in prep for migration

[bagg3rs]

Idea Description

We have lots of buckets of data which will need to be migrated or archived, this looks to understand the current data and then migrating look at migrating it.

Why Should We Do This

Discover PII - Implement a Lambda function or a Glue job to scan files for PII using AWS Macie or a custom regex-based solution.
Classify Data - Automatically tag data based on content sensitivity and domain relevance (e.g., finance, customer data, etc.).

Definition of Done

• Source bucket(s)/database access identified for a domain e.g. from here
• Data migration - cross-account region S3 Copy using some services script/lambda/batch operation to copy data from the source S3 buckets to the destination S3 buckets a (new) AWS account in London from Ireland
• Based on the classification, route/apply metadata to appropriate destination account based on domain-specific requirements. e.g. - raw, curated, standardised, derived/conformed, analysed

What skills do we need?

The usual aws. terra and then an interest to lean some other things maybe macie, Glue Data Quality

Notes

No response