Feature: Notifications verus polling

[gedw99]

Feature description

Hey

This plugin uses polling. Would be interesting if it could support Notifications.

AWS S3

https://docs.aws.amazon.com/AmazonS3/latest/userguide/EventNotifications.html

Minio

https://min.io/docs/minio/linux/administration/monitoring/bucket-notifications.html

Google Cloud Storage

https://github.com/conduitio-labs/conduit-connector-google-cloudstorage

What are the things to think about if we did want to do this ?

On receiving the Notification we could then do what the poller did, and so the OpenCDC output would be the same. So no changes to any common code in the Conduit system.

Not all S3 systems support it perhaps, but we could do it on a case by case basis.

The Notification system might not be a CloudEvent.

• AWS S3 can output to Amazon EventBridge, which is kind of Neural.
• Mini can do a Web Hook.

Notification format conversion. Each Provider has a different way of describing the Notification, and so we would have to convert them. We are only interested in the File change Delta, and not things like Security aspects liked ACL's.

[raulb]

👋 Thank you @gedw99 for showing your interest in making this connector more flexible. I agree that having the connector able to subscribe to these notifications would make it more efficient. I can't make promises on how soon we'll get this done since we have other priorities in the works, but we'll keep this one on our radar for future implementation. If you're willing to open a pull request implementing this, we'd be happy to assist with anything needed there.

Thank you again!