1.6.0 is here! and we finally have observability...

[ivanleomk]

We’re releasing instructor 1.6.2 today!

A huge change as we move from 1.5 -> 1.6 was the refactoring of our internal codebase itself. This should make it significantly easier to add support for new clients in the future with the more declarative logic. In this announcement article, we'll talk a bit about the new changes that we've made to the library as well as highlight some recent clients that we've added support for. We'll cover

1. Greater support for logging through the hook mechanism
2. New support for Cerebras
3. A brand new Fireworks client

Hooks

To see the documentation behind how to use this new feature, check out our documentation on Hooks here.

A major point of concern for a long time was observability in instructor. Our default suggestion was to use something like Langsmith or Logfire to handle logging of user requests and handle observability.

But with instructor 1.6, you're now able to add your own python functions to log and handle completions generated by your language model at five distinct stages.

1. completion:kwargs : This is when we recieve the completion arguments
2. completion:response : This is when we recieve the response from a language model
3. completion:error : This happens when we encounter an error during completion ( Normally thrown by the inference provider itself )
4. parse:error : This happens when we encounter an error during the parsing of the completion response
5. completion:last_attempt : This is when the last retry attempt is made.

This is done by using the .on method that's available for every client.

Let's see this in action.

import instructor
import openai
import pprint

client = instructor.from_openai(openai.OpenAI())


def log_completion_kwargs(*args, **kwargs):
    pprint.pprint({"args": args, "kwargs": kwargs})


client.on("completion:kwargs", log_completion_kwargs)

resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello, world!"}],
    response_model=str,
)
print(resp)
#> Hello! How can I assist you today?

When we use this new log_completion_kwargs method above, what we'll see in the console will be

{'args': (),
 'kwargs': {'messages': [{'content': 'Hello, world!', 'role': 'user'}],
            'model': 'gpt-3.5-turbo',
            'tool_choice': {'function': {'name': 'Response'},
                            'type': 'function'},
            'tools': [{'function': {'description': 'Correctly Formated and '
                                                   'Extracted Response.',
                                    'name': 'Response',
                                    'parameters': {'properties': {'content': {'title': 'Content',
                                                                              'type': 'string'}},
                                                   'required': ['content'],
                                                   'type': 'object'}},
                       'type': 'function'}]}}

You can add as many hooks as you'd like during each phase as long as the function names are unique. You can then remove it by using

client.off(<hookname>, <function name>)
client.clear() # This clears everything

Cerebras Support

See a longer write up here on Cerebras and how to use it here

We've added new support for the Cerebras client and with it comes blazing fast inference speeds - 20x faster than on GPUs in fact with 500 tok/s of inference speeds.

cerebras.mp4

To use it, just use the from_cerebras wrapper as seen below.

import instructor
from cerebras.cloud.sdk import Cerebras
from pydantic import BaseModel

client = instructor.from_cerebras(Cerebras())


class Person(BaseModel):
    name: str
    age: int


resp = client.chat.completions.create(
    model="llama3.1-70b",
    messages=[
        {
            "role": "user",
            "content": "Extract the name and age of the person in this sentence: John Smith is 29 years old.",
        }
    ],
    response_model=Person,
)

print(resp)
#> Person(name='John Smith', age=29)

Similar to our other clients, we also support streaming but only for text completions for now using the CEREBRAS_JSON mode.

Fireworks Client

We've also dropped a new fireworks client that provides integrated streaming of tool calls, structured outputs and more! This means that you'll be able to use models such as the phi-3-vision-instruct to do structured tool calling on.

Let's take this image below

import instructor
from fireworks.client import Fireworks
from pydantic import BaseModel

client = instructor.from_fireworks(Fireworks())


class Description(BaseModel):
    description: str
    country: str


image = instructor.Image.from_url(
    "https://images.unsplash.com/photo-1582538885592-e70a5d7ab3d3?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=1770&q=80"
)

resp = client.chat.completions.create(
    model="accounts/fireworks/models/phi-3-vision-128k-instruct",
    messages=[
        {
            "role": "user",
            "content": ["What's in this image?", image],
        }
    ],
    response_model=Description,
)

print(resp)
# > description='A view of a famous landmark framed by pink cherry blossoms. The sky is blue and partly cloudy.' country='Japan'

You also get the full benefits of streaming with our new Fireworks integration for both tool calling and text completions ( whichever works better for you!) as seen below. This is a huge improvement if you're building tools where you might want to render text content or UIs as they stream in for quick user responsiveness.

Partial.mp4