[spike] pluggable datastores for CDP

[mollykarcher]

What problem does your feature solve?

Ultimately, we want to make it as easy as possible for external contributors to create data lakes using different technologies (s3, r2, mongo, mqtt, etc). With how the code is currently structured, if someone were to create a new data storage option, they would have to contribute it back to this repository (go monorepo). This makes it so that the responsibility for quality and future maintenance for all data stores technically lies with the maintainers of this repo (SDF). We don't want to slow things down by putting ourselves in the middle here, and we don't want to be the arbiters of what people can build and how they build it.

What would you like to see?

A spike or design proposal that outlines how we could restructure our code or repositories in a way that would allow Galexie to accept pluggable datastores. This could mean that the interface for how to create a datastore is public, and in some separate repo that is maintained in an SDF-owned repo, but the implementations live in different, disperse repos. Also keep in mind that we already want to pull out the consumption components of CDP (#5525) into their own repo.

The "dream" dev journey could look something like:

• Implement some interface for a datastore in my own github repository
• Download/install Galexie, and it's configuration accepts my pluggable datastore interface/config with no code changes to Galexie necessary
• Pull the ingest SDK in my language of choice, and it's configuration accepts my pluggable datastore config with no code changes necessary