Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(ai-proxy): add AWS Bedrock Converse-API Driver #13354

Merged
merged 24 commits into from
Jul 30, 2024
Merged

feat(ai-proxy): add AWS Bedrock Converse-API Driver #13354

merged 24 commits into from
Jul 30, 2024

Conversation

tysoekong
Copy link
Contributor

Summary

Supersedes #13054 which was completely broken.

Adds AWS Bedrock "Converse API" support to Kong AI Gateway.

Checklist

  • The Pull Request has tests
  • A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
  • There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

AG-14

@subnetmarco
Copy link
Member

@fffonion let's start the review process on this large PR.

fffonion
fffonion reviewed Jul 22, 2024
View reviewed changes
kong/llm/drivers/bedrock.lua Show resolved Hide resolved
kong/llm/drivers/bedrock.lua Outdated Show resolved Hide resolved
kong/llm/init.lua Show resolved Hide resolved
kong/llm/init.lua Outdated Show resolved Hide resolved
kong/tools/aws_stream.lua Show resolved Hide resolved
@tysoekong tysoekong force-pushed the feat/ai_proxy_aws_bedrock branch from 2c5883a to 1a541a3 Compare July 24, 2024 15:30
@tysoekong tysoekong force-pushed the feat/ai_proxy_aws_bedrock branch 3 times, most recently from fa52c16 to f499d3f Compare July 25, 2024 13:02
@tysoekong tysoekong changed the title Feat/ai proxy aws bedrock feat: AI Proxy, AWS Bedrock Converse API Driver Jul 25, 2024
@tysoekong tysoekong changed the title feat: AI Proxy, AWS Bedrock Converse API Driver feat: AI Proxy, AWS Bedrock Converse AI Driver Jul 25, 2024
jschmid1
jschmid1 reviewed Jul 25, 2024
View reviewed changes
Copy link
Contributor

@jschmid1 jschmid1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1/n

kong/clustering/compat/checkers.lua Outdated Show resolved Hide resolved
kong/clustering/compat/checkers.lua Outdated Show resolved Hide resolved
kong/llm/drivers/shared.lua Show resolved Hide resolved
@tysoekong tysoekong force-pushed the feat/ai_proxy_aws_bedrock branch from 9d864e7 to f704158 Compare July 25, 2024 19:12
@tysoekong tysoekong changed the title feat: AI Proxy, AWS Bedrock Converse AI Driver feat: AI Proxy, AWS Bedrock Converse-API Driver Jul 25, 2024
windmgc
windmgc reviewed Jul 26, 2024
View reviewed changes
kong/plugins/ai-proxy/handler.lua Outdated Show resolved Hide resolved
kong/plugins/ai-proxy/handler.lua Outdated Show resolved Hide resolved
kong/plugins/ai-proxy/handler.lua Show resolved Hide resolved
kong/plugins/ai-proxy/handler.lua Outdated Show resolved Hide resolved
fffonion
fffonion reviewed Jul 26, 2024
View reviewed changes
kong/llm/drivers/bedrock.lua Outdated Show resolved Hide resolved
kong/llm/drivers/gemini.lua Outdated Show resolved Hide resolved
kong/llm/drivers/shared.lua Outdated Show resolved Hide resolved
kong/llm/schemas/init.lua Outdated Show resolved Hide resolved
@tysoekong
Copy link
Contributor Author

@fffonion I done all your second round comments

@fffonion fffonion changed the title feat: AI Proxy, AWS Bedrock Converse-API Driver feat(ai-proxy): add AWS Bedrock Converse-API Driver Jul 29, 2024
@fffonion
Copy link
Contributor

LGTM, please make sure comments from other reviewers are addressed : )

tysoekong and others added 19 commits July 29, 2024 15:38
@tysoekong
add complex provider signing support to ai-transformer plugin functions
2d04d68
@tysoekong
add bedrock provider options to shared schema
e21d531
@tysoekong
add aws provider to keybastion loose table
14e6fba
@tysoekong
adjust sdk to use new streaming parser format
1942f0a
@tysoekong
add bedrock format test scaffolding
810891e
@tysoekong
add bedrock driver class itself
c037480
@tysoekong
add bedrock drivers to rockspec
0cb31f2
@tysoekong
fix(ai-proxy)(general): body_filter should not run on failure
9252142
@tysoekong
add aws streaming format parser test
7275453
@tysoekong
feat(ai-proxy): bedrock unit tests
152b98c
@tysoekong
fix(ai-proxy): fix gemini streaming; fix gemini analytics
d044735
@tysoekong
fix(ai-proxy): bedrock: cluster compat
154d31d
@tysoekong
Update bedrock.lua
5e00426
@tysoekong
fix(clustering): faster comparisons for unsupported AI providers
da4fcab
@tysoekong
fix(clustering): faster comparisons for unsupported AI providers
6826842
@tysoekong
fix(ai-proxy): bedrock: refactor auth chain
dcd1d21
@tysoekong
lint
8d4ded7
@tysoekong
fix(ai-proxy): (bedrock, gemini): comments rollup
334df7f
@tysoekong @windmgc
Update kong/plugins/ai-proxy/handler.lua
8184ee0 
Co-authored-by: Keery Nie <[email protected]>
@tysoekong tysoekong force-pushed the feat/ai_proxy_aws_bedrock branch from 1b85373 to 8184ee0 Compare July 29, 2024 14:38
@fffonion fffonion merged commit 264341d into master Jul 30, 2024
25 checks passed
@fffonion fffonion deleted the feat/ai_proxy_aws_bedrock branch July 30, 2024 07:48
@team-gateway-bot
Copy link
Collaborator

Cherry-pick failed for master, because it was unable to cherry-pick the commit(s).

Please cherry-pick the changes locally.

git remote add upstream https://github.com/kong/kong-ee
git fetch upstream master
git worktree add -d .worktree/cherry-pick-13354-to-master-to-upstream upstream/master
cd .worktree/cherry-pick-13354-to-master-to-upstream
git checkout -b cherry-pick-13354-to-master-to-upstream
ancref=$(git merge-base 61e2c761d7ec7757552b54851a992b5a8740f4be 8184ee0faaee6adf6ff71de0f8f5ce88fbfa326e)
git cherry-pick -x $ancref..8184ee0faaee6adf6ff71de0f8f5ce88fbfa326e

@github-actions github-actions bot added the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Jul 30, 2024
@fffonion
Copy link
Contributor

fffonion commented Jul 30, 2024 •
edited
Loading

cc @ttyS0e I will do the cherrypicks to EE because there are couple of PRs that needs to cooridnate for conflict.

@fffonion
Copy link
Contributor

https://github.com/Kong/kong-ee/pull/9824

@AndyZhang0707 AndyZhang0707 removed the incomplete-cherry-pick A cherry-pick was incomplete and needs manual intervention label Aug 1, 2024
@randmonkey randmonkey mentioned this pull request Aug 5, 2024
Closed
1 task
oowl pushed a commit that referenced this pull request Aug 15, 2024
@fffonion @tysoekong
feat(ai-proxy): add AWS Bedrock Converse-API Driver (#13354) (#9824)
61fc9ea 
Supersedes #13054 which was completely broken.

Adds AWS Bedrock "Converse API" support to Kong AI Gateway.

AG-14

Co-authored-by: Jack Tysoe <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ttyS0e ttyS0e ttyS0e left review comments

@oowl oowl oowl left review comments

@jschmid1 jschmid1 jschmid1 left review comments

@windmgc windmgc windmgc approved these changes

@fffonion fffonion fffonion approved these changes

Assignees

@tysoekong tysoekong

Labels
cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee core/clustering plugins/ai-proxy plugins/ai-request-transformer plugins/ai-response-transformer schema-change-noteworthy size/XXL
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
9 participants
@tysoekong @subnetmarco @fffonion @team-gateway-bot @windmgc @jschmid1 @oowl @ttyS0e @AndyZhang0707