[BUG] Instructor Instrumentator causing Pydantic ValidationErrors

[ikarth]

Describe the bug
I'm having repeated validation errors when I'm attempting to use the InstructorInstrumentor (via LiteLLM talking to vLLM).

To Reproduce

import instructor
from litellm import completion, drop_params
import litellm
litellm.drop_params = True

from typing import Iterable, List, Optional
from pydantic import BaseModel, ConfigDict, Field

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

endpoint = "http://127.0.0.1:6006/v1/traces"
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))
from openinference.instrumentation.instructor import InstructorInstrumentor
from openinference.instrumentation.litellm import LiteLLMInstrumentor

InstructorInstrumentor().instrument(tracer_provider=tracer_provider)
LiteLLMInstrumentor().instrument(tracer_provider=tracer_provider)

client = instructor.from_litellm(completion)

class WeatherReport(BaseModel):
	"""The weather for today."""
	temperature: float
	precipitation: str

messages = [{"role": "user", "content": "Generate a weather report."}]

response = client.chat.completions.create(
		api_base=API_BASE,
        api_key=API_KEY,
        model=API_MODEL,
        messages=messages,
        response_model=WeatherReport
		)

print(response)

Results in a Pydantic ValidationError:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/luxor/src/lamesh/test_tele.py", line 39, in <module>
    response = client.chat.completions.create(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/instructor/client.py", line 172, in create
    return self.create_fn(
           ^^^^^^^^^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/instructor/_wrappers.py", line 113, in patched_new_func
    with self._tracer.start_as_current_span(
  File "/miniconda3/envs/luxor/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/config.py", line 396, in start_as_current_span
    span = self.start_span(
           ^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/config.py", line 439, in start_span
    span.set_attributes(attributes)
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/config.py", line 334, in set_attributes
    self.set_attribute(k, v)
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/config.py", line 341, in set_attribute
    value = self._self_config.mask(key, value)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/openinference/instrumentation/config.py", line 273, in mask
    return value() if callable(value) else value
           ^^^^^^^
  File "/miniconda3/envs/luxor/lib/python3.12/site-packages/pydantic/main.py", line 212, in __init__
    validated_self = self.__pydantic_validator__.validate_python(data, self_instance=self)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
pydantic_core._pydantic_core.ValidationError: 2 validation errors for WeatherReport
temperature
  Field required [type=missing, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing
precipitation
  Field required [type=missing, input_value={}, input_type=dict]
    For further information visit https://errors.pydantic.dev/2.9/v/missing

For some reason, moving the client = instructor.from_litellm(completion) line above the telemetry setup gives a different warning:

import instructor
from litellm import completion, drop_params
import litellm
litellm.drop_params = True

client = instructor.from_litellm(completion)

from typing import Iterable, List, Optional
from pydantic import BaseModel, ConfigDict, Field

from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor

endpoint = "http://127.0.0.1:6006/v1/traces"
tracer_provider = TracerProvider()
tracer_provider.add_span_processor(SimpleSpanProcessor(OTLPSpanExporter(endpoint)))
from openinference.instrumentation.instructor import InstructorInstrumentor
from openinference.instrumentation.litellm import LiteLLMInstrumentor

InstructorInstrumentor().instrument(tracer_provider=tracer_provider)
LiteLLMInstrumentor().instrument(tracer_provider=tracer_provider)


class WeatherReport(BaseModel):
	"""The weather for today."""
	temperature: float
	precipitation: str

messages = [{"role": "user", "content": "Generate a weather report."}]

response = client.chat.completions.create(
		api_base=API_BASE,
        api_key=API_KEY,
        model=API_MODEL,
        messages=messages,
        response_model=WeatherReport
		)

print(response)

Invalid type ModelMetaclass in attribute 'output.value' value sequence. Expected one of ['bool', 'str', 'bytes', 'int', 'float'] or None
temperature=23.0 precipitation='light rain'

Expected behavior
The expected response is that the validation would work after the generation is finished.

temperature=23.0 precipitation='light rain'

Environment (please complete the following information):

• OS: Windows (Ubuntu 22 via WSL)

arize-otel                               0.5.3
arize-phoenix                            5.7.0
arize-phoenix-evals                      0.16.1
arize-phoenix-otel                       0.6.1

instructor                               1.6.3

litellm                                  1.51.0

openinference-instrumentation            0.1.18
openinference-instrumentation-dspy       0.1.13
openinference-instrumentation-instructor 0.1.2
openinference-instrumentation-litellm    0.1.5
openinference-semantic-conventions       0.1.12
opentelemetry-api                        1.27.0
opentelemetry-exporter-otlp              1.27.0
opentelemetry-exporter-otlp-proto-common 1.27.0
opentelemetry-exporter-otlp-proto-grpc   1.27.0
opentelemetry-exporter-otlp-proto-http   1.27.0
opentelemetry-instrumentation            0.48b0
opentelemetry-proto                      1.27.0
opentelemetry-sdk                        1.27.0
opentelemetry-semantic-conventions       0.48b0

vllm                                     0.6.3.post1

[mikeldking]

Thanks for the note @ikarth ! We've tagged the maintainer of the instrumentation. Let us know if you need anything!

[glebmezh]

Ran into the same issue using Instructor + Anthropic Bedrock as the provider.

[mikeldking]

@ikarth - sorry you are facing issues with instructor - we added instrumentation prior to structured data output with the foundation models. If you are still using instructor and would like instrumentation, please let us know we can get this in better shape.

Sorry about the error - we will do better here.

[ikarth]

Still getting this error, I'm afraid.

[mikeldking]

Hey @ikarth so sorry this instrumentation has fallen out of a level of confidence. We're actively working on resourcing for instructor and will keep this ticket up to date as it gets assigned out.