Tools and structured output at the same time

[bytecod3r]

I'm curious if it's possible to have both structured output and tools work together, similar to querying OpenAI's GPT-4o-2024-08-6, where you provide a response schema and tools, allowing the LLM to decide whether to call the tools or respond using the defined schema.

I attempted a similar approach with the Instructor model, but it never called any tools. I also experimented with different modes, but without success.

anthropic_client = instructor.from_anthropic(Anthropic(api_key=self.api_key), mode=instructor.Mode.ANTHROPIC_TOOLS)

completion = anthropic_client.chat.completions.create(
    model=self.model,
    max_tokens=4096,
    stream=False,
    messages=[{
        "role": "user",
        "content": "add a new random task using add_tasks tool",
    }],
    response_model=schema,
    tools=tools
)

Any help is appreciated.