Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add caching options for system messages and tools in Anthropic LLM #1451

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

leecjohnny
Copy link

@leecjohnny leecjohnny commented Feb 5, 2025

TL;DR

Added caching support for system messages and tool definitions in the Anthropic LLM plugin.

What changed?

  • Added cache_system and cache_tools boolean options to LLMOptions
  • Implemented caching for system messages by adding cache control parameters when enabled
  • Added cache control support for tool/function definitions
  • Modified _latest_system_message and _build_function_description to handle caching configuration

How to test?

  1. Initialize the Anthropic LLM with caching enabled:
llm = LLM(..., cache_system=True, cache_tools=True)
  1. Run chat interactions with system messages and tool definitions
  2. Verify that system messages and tools are properly cached in the Anthropic API responses

Why make this change?

Caching system messages and tool definitions can improve performance and reduce token usage by allowing the Anthropic API to optimize repeated content across requests. This is particularly beneficial in long-running conversations or when using the same tools across multiple interactions.

Copy link

changeset-bot bot commented Feb 5, 2025

⚠️ No Changeset found

Latest commit: 8ee6b54

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

@CLAassistant
Copy link

CLAassistant commented Feb 5, 2025

CLA assistant check
All committers have signed the CLA.

@davidzhao davidzhao requested a review from jayeshp19 February 6, 2025 20:48
@davidzhao
Copy link
Member

thanks for the PR, there are a couple of tests that are failing, @leecjohnny could you take a look?

the ruff errors could be fixed with ruff format . && ruff check --fix .

@leecjohnny leecjohnny force-pushed the livekit-plugins-anthropic/prompt-caching branch from e33d416 to fe0a614 Compare February 6, 2025 21:02
@leecjohnny leecjohnny force-pushed the livekit-plugins-anthropic/prompt-caching branch from fe0a614 to 8ee6b54 Compare February 6, 2025 21:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants