Add custom partitioner in the producer?

[joelmunoz]

Currently, there is not way to add a custom partitioner using modern-cpp-kafka. What should I do in order to add this feature? Or Is there any release where this feature is available?

[kenneth-jia]

Actually, you're free to implement your customized partitioner and set it for a ProducerRecord (before sending the record). https://github.com/morganstanley/modern-cpp-kafka/blob/main/include/kafka/ProducerRecord.h#L43

[joelmunoz]

However, it does not work as

RdKafka::Conf* conf = RdKafka::Conf::create(RdKafka::Conf::CONF_GLOBAL);
CustomPartitioner partitioner;
conf->set("partitioner_cb", &partitioner, errstr);

In this way, the number of partitions of the target topic is provided by library itself (Java version uses a similar mechanism). On the other hand, using the ProducerRecord, I have to fetch topics metada in order to provide that information.

[kenneth-jia]

On the other hand, using the ProducerRecord, I have to fetch topics metada in order to provide that information

That's a good point!

[joelmunoz]

@kenneth-jia That is the reason I want to add this feature. I want to use the same idea used for "stats_cb", but for "partitioner_cb". However, before to develop it, I wanted to ask if you, as dev & maintainer, suggest another path.