Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Lambda model runner #331

Open
achad4 opened this issue Nov 8, 2024 · 0 comments
Open

[Feature] Lambda model runner #331

achad4 opened this issue Nov 8, 2024 · 0 comments

Comments

@achad4
Copy link

achad4 commented Nov 8, 2024

While LLMs will not be hosted directly on lambdas, allowing customers to proxy their eval requests through Lambda invocations would expand the breadth of use cases of FMEval, without needing to maintain a vast set of model runner classes. Use cases include:

  1. Use cases where customers host LLMs hosted on bare metal (EC2, etc).
  2. Use cases where LLMs that are hosted on Sagemaker or Bedrock, but invoked through a web service that performs preprocessing / post-processing.

The Lambda model runner can function identically to the Bedrock model runner, replacing the Bedrock runtime call with a lambda invocation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant