feat: Experiment and dataset improvements #6163
Conversation
dosubot
bot
added
size:XS
anticorrelator
changed the title
axiomofjoy
changed the title
| if span_kind == LLM: | ||
| return _get_llm_span_input( | ||
| input_messages=input_messages, | ||
| input_value=input_value, | ||
| input_mime_type=input_mime_type, | ||
| prompt_template_variables=prompt_template_variables, | ||
| tool_definition=tool_definition, |
There was a problem hiding this comment.
This value refers to an attribute on tool spans.
There was a problem hiding this comment.
I'm pretty sure these are defined under tools on llm spans, I adjusted this to reflect the right structure
anticorrelator
force-pushed
the
dustin/propagate-metadata
branch
from
7aecb06 to
6d59100
Compare
| if span_kind == LLM: | ||
| return _get_llm_span_input( | ||
| input_messages=input_messages, | ||
| input_value=input_value, | ||
| input_mime_type=input_mime_type, | ||
| prompt_template_variables=prompt_template_variables, | ||
| tool_definitions=tool_definitions, |
There was a problem hiding this comment.
can we just double check that this works with openai ft format? also it might be just simpler if the key was tools
| if tool_definitions_data := [ | ||
| _safely_json_decode(tool_definition) for tool_definition in tool_definitions | ||
| ]: | ||
| input["tool_definitions"] = tool_definitions_data |
There was a problem hiding this comment.
| input["tool_definitions"] = tool_definitions_data | |
| input["tools"] = tool_definitions_data |
Kinda leaning this direction. Does this work with openai ft / evals format?
There was a problem hiding this comment.
looks like the tool definitions are a key alongside the input:
https://platform.openai.com/docs/guides/fine-tuning#preparing-your-dataset-for-dpo
| @@ -795,7 +795,7 @@ async def chat_completion_create( | |||
| elif isinstance(event, anthropic_streaming.InputJsonEvent): | |||
| raise NotImplementedError | |||
| else: | |||
| assert_never(event) | |||
| assert_never(event) # type: ignore | |||
There was a problem hiding this comment.
There's a fix for this that @RogerHYang put out. It's the anthropic citations event. Can you cherry pick that and remove this ignore?
dosubot
bot
added
size:L
resolves #6082
resolves #6153
resolves #6139