Proposal for splitting metrics-reporter into modules

[mimaison]

This proposes splitting the metrics-reporter project into client and server modules so it's easier to evolve in the future.

[fvaleri]

Thanks @mimaison. I don't see a huge impact on the ongoing efforts, assuming this proposal is approved and we will have a release soon after. Can you clarify this in the proposal?

[fvaleri]

Why not KafkaClientMetricsReporter, KafkaServerMetricsReporter and YammerServerMetricsReporter? It just reads better.

[mimaison]

Happy to change the names if people prefer your suggestions. The original names also included Prometheus, I decided to drop it to have shorter names, but some people may prefer keeping that.

[scholzj]

Maybe we can add prometheus to the package name? That would keep E.g. io.strimzi.kafka.metrics.prometheus.ClientMetricsReporter. (I suspect we might need other technologies as well one (maybe not so distant) day :-/)

[fvaleri]

we might need other technologies as well

In that case, I would expect a config parameter to configure the output format with a default value set to Prometheus for backwards compatibility.

[mimaison]

If we are to add extra reporters in the future, I expect users would pick the format by using different classes. For example if you use io.strimzi.kafka.metrics.prometheus.ClientMetricsReporter the export format is Prometheus. So I wouldn't add a new configuration to the reporter itself.

[scholzj]

You should probably clarify what exactly are you talking about.

[ppatierno]

I assume that @mimaison metns that instead of having io.strimzi.metrics-reporter-0.1.0.jar as today, we would have io.strimzi.prometheus.metrics-reporter-0.2.0.jar instead. In the future, a different implementation would have a separate artifact with a different name (replacing "prometheus" with something else). This way if you want to use just the prometheus one you can get only the specific dependency and not bringing with you stuff which is unrelated but belongs to a different implementation.

[scholzj]

I would not expect different modules for different implementations. So I would avoid this.

[scholzj]

In any case if you decide to include Prometheus in the module name, it should absolutely not be in the group ID but only in the artifact ID.

[im-konge]

I guess that nobody knows how many dependencies would be useful or useless in different implementations. Also in what point in the future should we add another implementation? Having the name in the artifacts and modules now would look like there is another JAR containing different implementation (even if there is just Prometheus one).

I'm fine with adding prometheus package. For changing the names of the module and artifact - it could be changed in the future when another implementations will be added and the split will be needed, right?

[scholzj]

Can you please clarify how this will work for Connect / MM2? They use a lot of clients. But they also have their own metrics (connectors and tasks and their state, etc.). So it seems like the client reporter is not enough to provide these, or?

[mimaison]

The Connect runtime and Streams are considered clients. To collect their metrics you would still follow the instructions from https://github.com/strimzi/metrics-reporter?tab=readme-ov-file#kafka-connect-and-kafka-streams but use io.strimzi.kafka.metrics.ClientMetricsReporter as the class name.

[scholzj]

So essentially, when used in Connect, the client reporter will automatically handle the additional Connect metrics as well?

[mimaison]

Yes, like io.strimzi.kafka.metrics.KafkaPrometheusMetricsReporter does at the moment.

[ppatierno]

AFAIK, there is Prometheus Agent which is able to use the push-based model so writing metrics remotely as well.
Regarding the naming I am fine with ClientMetricsReporter but maybe within a prometheus package as suggested. It would be clearly Kafka even because the FQDM has kafka in the package as well.

[tombentley]

LGTM, thanks @mimaison

[scholzj]

Thanks @mimaison. +1 binding.

[ppatierno]

LGTM. Thanks Mickael!
When this proposal is approved and merged we should try to have the implementation and the release done before a new operator and bridge release so that we don't need to make breaking changes as described in your compatibility section imho.

[mimaison]

I have most of the code ready, so I should be able to open a PR next week.

[fvaleri]

Thanks.