Debezium vs Conduit: Differences and Recommendations

[AndryHTC]

Hi all,

I've been exploring real-time data integration tools recently and have come across both Debezium and Conduit. I understand that both projects are designed to listen to changes in databases (among other sources) and propagate those changes to downstream consumers. However, I would love to better understand the differences between these two tools and the use cases for which each is best suited.

From my research, I've gathered that:

• Debezium is a mature project built in Java and on top of Apache Kafka. It has a robust community and is widely adopted.
• Conduit is a relatively new project written in Go, designed to be simple, efficient, and extensible. It runs pipelines in their own goroutines and doesn't require additional components like a Kafka cluster.

I'm particularly interested in understanding more about the following:

1. Performance Differences: What are the performance characteristics of each system? In what scenarios would one tool outperform the other?

2. Ease of Use: How does the user experience of both tools compare? How does Conduit's out-of-the-box UI compare with the tooling available for Debezium?

3. Connectivity: How does the extensibility and connector ecosystem of each tool compare? How straightforward is it to add custom connectors in each system?

4. Scalability and Resilience: How do the two systems compare in terms of scaling horizontally and recovering from failures? What kind of traffic volumes can each system handle?

5. Community and Support: How mature and responsive is the Conduit community compared to the Debezium community?

I would greatly appreciate any insight or experiences you could share about these two (or maybe more) technologies and how to choose between them.

Thank you!

[lovromazgon]

Hi @AndryHTC!

I'm happy to see you're interested in Conduit. I can try to help you with your inquiry about Conduit and Debezium, but keep in mind that as a Conduit dev, I am biased towards Conduit, and I wouldn't call myself a Debezium expert.

Performance Differences: Conduit is written in Go and is therefore much more lightweight compared to Debezium. It heavily utilizes goroutines, so running Conduit on multi-core machines should allow it to parallelize its workload. We have not done any direct comparisons in throughput though. It's hard to make any promises regarding throughput as it depends on lots of factors like the hardware specifications, the source and destination resources, the size of the records and transforms applied to them (to name a few). That said, to illustrate what you can expect I tried running the example pipeline and averaged a throughput of over 30k msg/s with an average latency of <1ms from read to write (end-to-end) on my MacBook.

Ease of Use: Conduit exposes an API (REST and gRPC) which can be leveraged to build your own interface for interacting with Conduit. It also comes with a pre-packaged UI for quickly getting started. We expect most users to use the UI for building pipelines during development. Provisioning pipelines in production should probably be done using pipeline configuration files and monitored using the exposed metrics. From the point of view of an operator, Conduit should be easier to set up, since it's essentially just a single binary that can run on all popular OS/Arch combinations without any external dependencies like Kafka or Zookeeper. This also means that, unlike Debezium, Conduit does not force you to include Kafka in your data pipelines, instead it allows you to push data from any source to any destination.

Connectivity: Conduit currently provides over 40 connectors (and more to come!). In addition to the native connectors built specifically for Conduit, there's also a Kafka Connect wrapper connector, which lets you run any Kafka Connect connector as a Conduit connector (including Debezium connectors). If you're interested in writing your own connectors, we recommend doing so using the Go Connector SDK. It should be as easy as creating a new repository using the connector template and implementing a few interfaces. Since the communication between the connector and Conduit happens through gRPC you can also develop a connector in a language other than Go, although we currently don't provide an SDK for other languages, so that will be a bit more difficult.

Scalability and Resilience: Conduit currently does not provide a mode to scale horizontally. However, there is a Kubernetes Operator in the works, which will ultimately allow you to provision pipelines on a cluster of Conduit instances and orchestrate failover behavior.

Community and Support: The Conduit community is still quite small compared to the Debezium community, but we are dedicated to steadily growing it and providing support (hint: join our Discord!). We use Conduit in production at Meroxa and depend heavily on it, so we will continue improving it for the foreseeable future.

At the end, I'd like to add that our engineers at Meroxa have a lot of experience running Debezium and Kafka Connect at scale. While Debezium is obviously the more mature project of the two, Conduit tries to grow from the experience gathered in the past and improve the user experience by applying the lessons learned.

[AndryHTC]

Honestly, I was hoping for a slightly biased comparison 😂, and you've delivered it flawlessly.

Based on your experience, are there any other newcomers in the industry attempting to replicate what Conduit is aiming to achieve?

I genuinely admire the project and appreciate how user-friendly it is for developers with limited prior knowledge 🤩.