Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding table for example metadata for experiments #147

Merged
merged 1 commit into from
Feb 21, 2025
Merged

Adding table for example metadata for experiments #147

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions website/docs/autoeval/experiments.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,16 @@ Anything that influences the LLM application's performance is measureable throug
- Adjusting **system prompts** for an agent
- And more

Here are the recommended metadata to track for your LLM application:
| | model version | chunk_size | chunk_strategy | top_k_retrieval | dataset | temperature |
| --------- | -------------------- | --------- | -------------- | --------------- | ------------------ | ----------- |
| Example 1 | gpt-4o-2024-08-06 | 1024 | sliding window | 5 | question-answer-v1 | 0 |
| Example 2 | gemini-2.0-flash-001 | 512 | no overlap | 5 | qusetion-answer-v1 | 0 |

:::info
For saving metadata around the model, make sure to include the entire version, which includes the cutoff of the train date. Model providers are constantly updating the latest model with new data, as such it's a good practice to pin the model version on the one that works best for you.
:::

## Usage Guide
This guide walks through the process of setting up and running experiments using AutoEval, including:
1. **Setting up the API key** and **creating a project**
Expand Down