Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bedrock-runtime client apply_guardrail function adds new line at the end of masked text #4317

Open
1 task
MottiG opened this issue Oct 26, 2024 · 1 comment
Assignees
Labels
bedrock-runtime bug This issue is a confirmed bug. p2 This is a standard priority issue service-api This issue is caused by the service API, not the SDK implementation.

Comments

@MottiG
Copy link

MottiG commented Oct 26, 2024

Describe the bug

When using bedrock-runtime apply_guardrail on a given text, where guardrail is configured with mask action, the returned text is indeed masked where needed, but a new line is added to the given input.

TEXT = "The official address of the UK Prime Minister\'s office is:\\n\\n10 Downing Street\\nLondon\\nSW1A 2AA\\nUnited Kingdom\\n\\nThis address is commonly referred to simply as \\"10 Downing Street\\" or \\"Number 10.\\" It has been the official residence and office of the British Prime Minister since 1735. The building is located in central London, near the Houses of Parliament and Whitehall."

bedrock = boto3.client(...).   # bedrock has a configured guardrails with "ADDRESS" PII and action "mask"
guard_res = bedrock.apply_guardrail(
    guardrailIdentifier=...,
    guardrailVersion=..., 
    source="OUTPUT",
    content=[{'text':{'text':TEXT}}]
)

Regression Issue

  • Select this option if this issue appears to be a regression.

Expected Behavior

Bedrock should return the text as is, while masking the configured addresses without a newline at the end.

guard_res['outputs'][0]['text'] should be:

"The official address of the UK Prime Minister\'s office is:\n\n{ADDRESS}\n{ADDRESS}\n\nThis address is commonly referred to simply as "{ADDRESS}." It has been the official residence and office of the British Prime Minister since 1735. The building is located in central {ADDRESS}, near the Houses of Parliament and Whitehall."

Current Behavior

Actual results (guard_res['outputs'][0]['text']):

"The official address of the UK Prime Minister\'s office is:\n\n{ADDRESS}\n{ADDRESS}\n\nThis address is commonly referred to simply as "{ADDRESS}." It has been the official residence and office of the British Prime Minister since 1735. The building is located in central {ADDRESS}, near the Houses of Parliament and Whitehall.\n"

As you can see when scrolling to the end of the resulting text, a newline character \n was added to the end of the text.

Reproduction Steps

TEXT = "The official address of the UK Prime Minister\'s office is:\\n\\n10 Downing Street\\nLondon\\nSW1A 2AA\\nUnited Kingdom\\n\\nThis address is commonly referred to simply as \\"10 Downing Street\\" or \\"Number 10.\\" It has been the official residence and office of the British Prime Minister since 1735. The building is located in central London, near the Houses of Parliament and Whitehall."

bedrock = boto3.client(...).   # bedrock has a configured guardrails with "ADDRESS" PII and action "mask"
guard_res = bedrock.apply_guardrail(
    guardrailIdentifier=...,  # add guardrail id
    guardrailVersion=...,  # add guardrail version or 'DRAFT'
    source="OUTPUT",
    content=[{'text':{'text':TEXT}}]
)

print(guard_res['outputs'][0]['text'].endswith('\n') is True)
print(guard_res['outputs'][0]['text'])

Possible Solution

I'm unsure of the root cause of this behavior, but I see that the client already got the new line char as a response to the API call.

Additional Information/Context

I also checked the usage of configuredGuardrail in the call for the converse function. The behavior didn't reproduce, and the guarded text returned as expected.

SDK version used

1.34.162

Environment details (OS name and version, etc.)

MacOS 14.7

@MottiG MottiG added bug This issue is a confirmed bug. needs-triage This issue or PR still needs to be triaged. labels Oct 26, 2024
@tim-finnigan tim-finnigan self-assigned this Oct 29, 2024
@tim-finnigan tim-finnigan added the investigating This issue is being investigated and/or work is in progress to resolve the issue. label Oct 29, 2024
@tim-finnigan
Copy link
Contributor

Thanks for reaching out. I could reproduce the behavior you described where \n was added to the text output when applying a guardrail with "Address" PII and action "mask". The apply_guardrail command makes a call to the underlying ApplyGuardrail API, so this is an issue with the Bedrock service API rather than with boto3 directly.

If you add boto3.set_stream_logger('') to your script then it will show the full debug logs and what the response looks like from the Bedrock service API. When doing that, I see the \n is in the API response body, which confirms that this is an issue with the API rather than the SDK. I'll go ahead and forward this issue to the Bedrock team internally for further review, and will share any updates here.

@tim-finnigan tim-finnigan added service-api This issue is caused by the service API, not the SDK implementation. p2 This is a standard priority issue bedrock-runtime and removed investigating This issue is being investigated and/or work is in progress to resolve the issue. needs-triage This issue or PR still needs to be triaged. labels Oct 29, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bedrock-runtime bug This issue is a confirmed bug. p2 This is a standard priority issue service-api This issue is caused by the service API, not the SDK implementation.
Projects
None yet
Development

No branches or pull requests

2 participants