Instructor failing with datetime task | Tested with Groq Llama | Working with LC pydanticOutputParser

[97k]

I have a simple response model and a simple task of converting natural language time to datetime.
I have tried playing with both 8B and 70B, it fails every single time, even after moderating the prompts.

It works sometimes with 8B though (1 out of 10 times)

In langchain, it is working as intended. I am curious to why Groq api returns 400 / BadRequest error only with instructor

Here's the code

from pydantic import BaseModel
from datetime import datetime


class TaskTimes(BaseModel):
    ''' Natural Language to Datetime'''
    times: list[datetime]

Instructor Code

time: TaskTimes = instrcutor_client.chat.completions.create(
  model=LLMModels.groq_llama_70b,
  response_model=TaskTimes,
  messages = [
        {"role": "system", "content": NATURAL_TIME_CONVERT_SYSTEM},
        {"role": "user", "content": f"Current time is {datetime.now()}, convert: today at 12pm}
  ]

This raises an error

BadRequestError: Error code: 400 - {'error': {'message': "Failed to call a function. Please adjust your prompt. See 'failed_generation' for more details.", 'type': 'invalid_request_error', 'code': 'tool_use_failed', 'failed_generation': '\n{\n\t"tool_calls": [\n\t\t{\n\t\t\t"id": "pending",\n\t\t\t"type": "function",\n\t\t\t"function": {\n\t\t\t\t"name": "TaskTimes"\n\t\t\t},\n\t\t\t"parameters": {\n\t\t\t\t"times": ["2024-06-13T12:00:00"]\n\t\t\t}\n\t\t}\n\t]\n}\n'}}

With Langchain's output parser

from langchain_groq import ChatGroq
from langchain_core.prompts import ChatPromptTemplate
from langchain.output_parsers import PydanticOutputParser

time_parser = PydanticOutputParser(pydantic_object=TaskTimes)
time_prompt = ChatPromptTemplate.from_messages(
    [
        (
            "system",
            "Convert this natural language time to date-time relative to current time. DO NOT generate preamble or explanation. The output should matches the following instruction {format_instruction} .",
        ),
        ("human", "Current time is {current_time}, convert {natural_time}"),
    ]
).partial(format_instruction=time_parser.get_format_instructions())

groq_lc = ChatGroq(model=LLMModels.groq_llama_70b, api_key=settings.GROQ_API_KEY.get_secret_value())

nat_datetime_lc = time_prompt | groq_lc | time_parser

Langchain output

nat_datetime_lc.invoke({"current_time": datetime.now(), "natural_time": "today at 12 pm"})

TaskTimes(times=[datetime.datetime(2024, 6, 13, 12, 0)])`

I have more than 20 components working with instructor and everytime they work. I am curious in why this is only happening in this case, everytime.

[97k]

Hey @jxnl
What could be the possible reason of this happening? This is surprising. Would you please share your thoughts?

[jxnl]

im not an expert at prompting 70b llama, instructor has fewer prompts so you'll need to prompt more to get good performance vs langchain's longer prompt

[97k]

the cookbooks has limited prompts, that made me write small prompts. But I understand, it also depends on the quality of LLM, for llama, it might needs good detailed prompts.

Thank you for taking time to reply. Means a lot :)