[MSK] Unable to Switch from Standard to Express Instance Type

[tbazen]

Describe the bug

When attempting to update the instance type of an Amazon MSK cluster from a standard instance type to an express broker instance type, it is not updated to the specified instance type.

Current Instance Type: kafka.m5.2xlarge
Target Instance Type: express.m7g.4xlarge

I used the update_broker_type API to perform the switch. While the API call completes successfully and returns a ClusterOperationArn, the result is unexpected.

After waiting for the operation to complete, the instance type changes to kafka.m7g.4xlarge, which is still within the standard broker family. The cluster does not transition to the express broker type as intended.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The instance type should transition from standard instance type to the express broker type express.m7g.4xlarge. Alternatively, the request should fail with a descriptive error message.

Current Behavior

The instance type updates to standard broker family kafka.m7g.4xlarge, instead of express broker type express.m7g.4xlarge . The cluster does not transition to the express broker type.

Reproduction Steps

Below is a snippet of the code used for the operation:

def update_to_express_broker_type(cluster_arn, express_broker_type):
    client = boto3.client("kafka")
    cluster_info = client.describe_cluster_v2(ClusterArn=cluster_arn)["ClusterInfo"]
    current_version = cluster_info["CurrentVersion"]
    response = client.update_broker_type(
        ClusterArn=cluster_arn,
        CurrentVersion=current_version,
        TargetInstanceType=express_broker_type,
    )
    return response

update_to_express_broker_type("<cluster_arn>", "express.m7g.4xlarge")

1. Use the provided Python snippet to invoke the update_broker_type API.
2. Monitor the progress of the cluster operation.
3. Observe the resulting instance type.

Possible Solution

No response

Additional Information/Context

• The API request appears to be completed successfully.
• The returned ClusterOperationArn suggests the operation has been accepted.

SDK version used

1.34.149

Environment details (OS name and version, etc.)

MacOS Sequoia

[tim-finnigan]

Thanks for reaching out. The update_broker_type command calls the UpdateBrokerType API, so this appears to be an issue with the Kafka service rather than Boto3 directly.

Could you share debug logs (with any sensitive info redacted) to help with further investigation? You can get those by adding boto3.set_stream_logger('') to your script. If necessary we can reach out internally to the Kafka team for further review.

Edit - upon asking about this issue internally, someone from the Kafka team noted that they don’t support moving between standard and express. I asked about if there is documentation on this, or if the API should return an error in this case. Waiting to hear more info. If you're still able to share debug logs please do so.