KAFKA-18060: new coordinator does not handle TxnOffsetCommitRequest with empty member id when using CONSUMER group #17914
Conversation
…th empty member id when using CONSUMER group
dajac
changed the title
|
@brandboat Thanks for the quick patch. Could you please add a description to the pull request? We usually also use it for the commit message. |
There was a problem hiding this comment.
@brandboat Thanks for the patch. I left some comments for consideration.
| // TxnOffsetCommitRequest versions v0-v2 do not include a member ID. | ||
| // And we can still send UNKNOWN_GENERATION_ID and UNKNOWN_MEMBER_ID through Producer#sendOffsetsToTransaction. | ||
| // Therefore, we should not throw exception in these cases. |
There was a problem hiding this comment.
I would simplify it a bit. What about The TxnOffsetCommit API does not require the member id, the generation id and the group instance id fields. Hence the are only validated if any of them is provided.?
| @@ -26,6 +26,7 @@ | |||
| import org.apache.kafka.common.message.ConsumerGroupDescribeResponseData; | |||
There was a problem hiding this comment.
Could you please extend unit tests in ConsumerGroupTest too?
| adminClient = cluster.admin() | ||
| adminClient.createTopics(singleton(new NewTopic(topic, 1, 1.toShort))) |
There was a problem hiding this comment.
You could actually use createTopic method from the parent class.
| // add this line to avoid error INVALID_TXN_STATE | ||
| producer.sendOffsetsToTransaction( | ||
| Collections.singletonMap(new TopicPartition("topic", 0), new OffsetAndMetadata(5)), | ||
| new ConsumerGroupMetadata(groupId)) |
There was a problem hiding this comment.
I suppose that you must do this in order to add the __consumer_offsets partition to the transaction.
There was a problem hiding this comment.
Personally, I prefer to build the AddOffsetsToTxnRequest explicitly. for example:
protected def addOffsetsToTxn(groupId: String,
producerId: Int,
producerEpoch: Short,
transactionalId: String,
version: Short): Unit = {
val request = new AddOffsetsToTxnRequest.Builder(new AddOffsetsToTxnRequestData()
.setTransactionalId(transactionalId)
.setProducerId(producerId).setProducerEpoch(producerEpoch).setGroupId(groupId)
).build(version)
val expectedResponse = new AddOffsetsToTxnResponseData()
val response = connectAndReceive[AddOffsetsToTxnResponse](request)
assertEquals(expectedResponse, response.data)
}
| Collections.singletonMap(new TopicPartition("topic", 0), new OffsetAndMetadata(5)), | ||
| new ConsumerGroupMetadata(groupId)) | ||
|
|
||
| // verify that the TXN_OFFSET_COMMIT request is processed correctly when member id is UNKNOWN_MEMBER_ID |
| producer.close() | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
While we are here, do you mind adding a test which verifies that the fields are correctly validated when provided?
| new ClusterConfigProperty(key = GroupCoordinatorConfig.OFFSETS_TOPIC_PARTITIONS_CONFIG, value = "1"), | ||
| new ClusterConfigProperty(key = GroupCoordinatorConfig.OFFSETS_TOPIC_REPLICATION_FACTOR_CONFIG, value = "1"), | ||
| new ClusterConfigProperty(key = TransactionLogConfig.TRANSACTIONS_TOPIC_PARTITIONS_CONFIG, value = "1"), | ||
| new ClusterConfigProperty(key = TransactionLogConfig.TRANSACTIONS_TOPIC_REPLICATION_FACTOR_CONFIG, value = "1"), |
There was a problem hiding this comment.
nit: You can define all the common configs at the class level in ClusterTestDefaults.
There are two issues in KAFKA-18060:
New coordinator can't handle the TxnOffsetCommitRequest with empty member id, and TxnOffsetCommitRequest v0-v2 do definitely has an empty member ID, causing ConsumerGroup#validateOffsetCommit to throw an UnknownMemberIdException. This prevents the old producer from calling sendOffsetsToTransaction. Note that TxnOffsetCommitRequest versions v0-v2 are included in KIP-896, so it seems the new coordinator should support that operations
The deprecated API Producer#sendOffsetsToTransaction does not use v0-v2 to send TxnOffsetCommitRequest with an empty member ID. Unfortunately, it has been released for a while. Therefore, the new coordinator needs to handle TxnOffsetCommitRequest with an empty member ID for all versions.
Taken from the two issues above, we need to handle empty member id in all API versions when new coordinator are dealing with TxnOffsetCommitRequest.
Committer Checklist (excluded from commit message)