Unnecessary Limitation on Stop Sequences in /converse API of Botocore

[zhaodong-wang]

Describe the bug

I have encountered an issue with the /converse API in the Botocore API specification, specifically regarding the limitation on stop sequences. According to the [API spec](

botocore/botocore/data/bedrock-runtime/2023-09-30/service-2.json

Line 1891 in dd0e94f

"max":4,

), the InferenceConfigurationStopSequencesList only supports stop sequences with up to 4 words. This limitation does not exist in the regular Anthropic API, which allows for more flexibility.

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

The /converse API should allow stop sequences with more than 4 words, similar to the regular Anthropic API, to provide more flexibility and avoid unnecessary constraints.

Current Behavior

The API currently restricts stop sequences to a maximum of 4 words, which is not aligned with the capabilities of the regular Anthropic API.

Reproduction Steps

1. Attempt to use the /converse API with a stop sequence containing more than 4 words.
2. Observe that the API does not accept the request due to the limitation.

Example code:

import boto3
from botocore.exceptions import ClientError

def main():
    # Initialize the Bedrock runtime client
    bedrock_client = boto3.client('bedrock-runtime')

    # Specify the model ID you want to use
    model_id = 'your-model-id'  # Replace with your specific model ID

    # Define the system prompt to set the context for the conversation
    system_prompts = [
        {
            "text": "You are an assistant that provides concise and accurate information."
        }
    ]

    # Define the user message to start the conversation
    user_message = {
        "role": "user",
        "content": [
            {
                "text": "Can you provide a brief overview of the Amazon Bedrock Converse API?"
            }
        ]
    }

    # Define the inference configuration with stop sequences
    inference_config = {
        "stopSequences": ["Thank you", "Goodbye", "Stop", "End", "Exit"]
    }

    # Combine the messages into a list
    messages = [user_message]

    try:
        # Call the Converse API
        response = bedrock_client.converse(
            modelId=model_id,
            system=system_prompts,
            messages=messages,
            inferenceConfig=inference_config
        )

        # Extract and print the assistant's response
        assistant_message = response['output']['message']
        for content in assistant_message['content']:
            if 'text' in content:
                print(f"Assistant: {content['text']}")

    except ClientError as e:
        print(f"An error occurred: {e.response['Error']['Message']}")

if __name__ == "__main__":
    main()

Possible Solution

Consider removing the 4-word restriction on stop sequences in the /converse API to align with the regular Anthropic API's capabilities.

Additional Information/Context

This limitation seems unnecessary and adds constraints that are not present in the regular Anthropic API. It would be beneficial to remove this restriction to enhance the API's usability and flexibility.

SDK version used

1.36.6

Environment details (OS name and version, etc.)

macOS 15.1.1 (24B91)

[khushail]

Hi @zhaodong-wang , thanks for reporting this. On checking the AWS Docs, the Bedrock Converse API has this limit mentioned as 4-

https://docs.aws.amazon.com/bedrock/latest/APIReference/API_runtime_InferenceConfiguration.html#bedrock-Type-runtime_InferenceConfiguration-stopSequences

stopSequences

    A list of stop sequences. A stop sequence is a sequence of characters that causes the model to stop generating the response.

    Type: Array of strings

    Array Members: Minimum number of 0 items. Maximum number of 4 items.

    Length Constraints: Minimum length of 1.

    Required: No

I am not sure what do you mean when you say Regular Enthropic model. I see that on AWS Docs page, there are 2 different variations of models being used with different stop_sequence limits-

1. Anthropic Text Completion API -https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-text-completion.html#model-parameters-anthropic-claude-text-completion-overview , NO Limit has been mentioned.
2. Anthropic Claude message API - https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-anthropic-claude-messages.html, mentions the limit as 8191.

So could you please share some more details on what limit you are referring to and share the relevant doc pls.

Thanks.