UI tools

[sobolk]

Problem/Changes

This PR adds "client tools" concept to conversation handler.

They way this works is:

1. Client side can send "client tools" configuration in the event. This propagates through AppSync to conversation handler.
2. "Client tool" purpose is to grab inputs from LLM and execute processing in UI (or whatever "client side" is).
3. Client tool use interrupts a tool use loop in bedrock adapter to yield control back to UI.

Out of scope

Bedrock API may respond with a tool use response that includes multiple tools.
It is unclear what exactly should we do if mixed tool kinds are requested, so for simplicity, for now if any of the tools is client tool we interrupt, ignore other tools and respond with client side tool usage blocks.
This is to enable exploration of real use cases with client side tools.

Validation

Added e2e tests and unit tests.

Checklist

• If this PR includes a functional change to the runtime behavior of the code, I have added or updated automated test coverage for this change.
• If this PR requires a change to the Project Architecture README, I have included that update in this PR.
• If this PR requires a docs update, I have linked to that docs PR above.
• If this PR modifies E2E tests, makes changes to resource provisioning, or makes SDK calls, I have run the PR checks with the run-e2e label set.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[changeset-bot]

🦋 Changeset detected

Latest commit: 5253ff9

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@aws-amplify/ai-constructs	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[sobolk]

This abstraction appeared in third place already in public API. Therefore, extracting a type.

[rtpascual]

Am I understanding correctly that the event messages are included in the response? If not, how are we sure the response will always include Seattle instead of something like The temperature is x?

[sobolk]

Am I understanding correctly that the event messages are included in the response?

They're not. It's just tool use request (i.e. ContentBlock.ToolUseMember blocks) being returned.

If not, how are we sure the response will always include Seattle instead of something like The temperature is x?

I have not noticed that Seattle is poor choice for assertion given it's part of request too.

The response really looks like this:

{
  "conversationId": "59524ac0-43cc-4b34-9c5a-5ff2ee2778d0",
  "sender": null,
  "id": "6a4ed385-f686-4a2f-bdc0-356a3584e19d",
  "updatedAt": "2024-08-13T18:15:36.733Z",
  "createdAt": "2024-08-13T18:15:36.733Z",
  "content": "[{toolUse={input={city=Seattle}, name=thermometer, toolUseId=tooluse_ubgklazyReS7717_AIf31w}}]",
  "associatedUserMessageId": "d226bf37-dda4-4a4e-aeb6-6de69029bb4e",
  "__typename": "ConversationMessageAssistantResponse"
}

I'll make the assertion better.