+ You can check out the Colab Notebook if you'd like to jump straight to the code:
+
+ 
+
+
+
+## Getting started
+
+First, ensure you have both `opik` and `dspy` installed:
+
+```bash
+pip install opik dspy
+```
+
+In addition, you can configure Opik using the `opik configure` command which will prompt you for the correct local server address or if you are using the Cloud platfrom your API key:
+
+```bash
+opik configure
+```
+
+## Logging DSPy calls
+
+To log a DSPy pipeline run, you can use the [`OpikCallback`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/dspy/OpikCallback.html). This callback will log each DSPy run to Opik:
+
+```python
+import dspy
+from opik.integrations.dspy.callback import OpikCallback
+
+project_name = "DSPY"
+
+lm = dspy.LM(
+ model="openai/gpt-4o-mini",
+)
+dspy.configure(lm=lm)
+
+
+opik_callback = OpikCallback(project_name=project_name)
+dspy.settings.configure(
+ callbacks=[opik_callback],
+)
+
+cot = dspy.ChainOfThought("question -> answer")
+cot(question="What is the meaning of life?")
+```
+
+Each run will now be logged to the Opik platform:
+
+
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/gemini.md b/apps/opik-documentation/documentation/docs/tracing/integrations/gemini.md
index e4392bb54d..f661246157 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/gemini.md
+++ b/apps/opik-documentation/documentation/docs/tracing/integrations/gemini.md
@@ -44,7 +44,7 @@ In order to configure Gemini, you will need to have:
Once you have these, you can set them as environment variables:
-```python
+```python pytest_codeblocks_skip="true"
import os
os.environ["GEMINI_API_KEY"] = "" # Your Google AI Studio Gemini API Key
@@ -76,6 +76,9 @@ response = litellm.completion(
If you are using LiteLLM within a function tracked with the [`@track`](/tracing/log_traces#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
```python
+from opik import track, opik_context
+import litellm
+
@track
def generate_story(prompt):
response = litellm.completion(
@@ -83,7 +86,7 @@ def generate_story(prompt):
messages=[{"role": "user", "content": prompt}],
metadata={
"opik": {
- "current_span_data": get_current_span_data(),
+ "current_span_data": opik_context.get_current_span_data(),
},
},
)
@@ -98,7 +101,7 @@ def generate_topic():
messages=[{"role": "user", "content": prompt}],
metadata={
"opik": {
- "current_span_data": get_current_span_data(),
+ "current_span_data": opik_context.get_current_span_data(),
},
},
)
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/groq.md b/apps/opik-documentation/documentation/docs/tracing/integrations/groq.md
index 984631bc4f..98fdd02e4a 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/groq.md
+++ b/apps/opik-documentation/documentation/docs/tracing/integrations/groq.md
@@ -76,6 +76,9 @@ response = litellm.completion(
If you are using LiteLLM within a function tracked with the [`@track`](/tracing/log_traces#using-function-decorators) decorator, you will need to pass the `current_span_data` as metadata to the `litellm.completion` call:
```python
+from opik import track, opik_context
+import litellm
+
@track
def generate_story(prompt):
response = litellm.completion(
@@ -83,7 +86,7 @@ def generate_story(prompt):
messages=[{"role": "user", "content": prompt}],
metadata={
"opik": {
- "current_span_data": get_current_span_data(),
+ "current_span_data": opik_context.get_current_span_data(),
},
},
)
@@ -94,11 +97,11 @@ def generate_story(prompt):
def generate_topic():
prompt = "Generate a topic for a story about Opik."
response = litellm.completion(
- model="Groq/Groq-pro",
+ model="groq/llama-3.3-70b-versatile",
messages=[{"role": "user", "content": prompt}],
metadata={
"opik": {
- "current_span_data": get_current_span_data(),
+ "current_span_data": opik_context.get_current_span_data(),
},
},
)
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/haystack.md b/apps/opik-documentation/documentation/docs/tracing/integrations/haystack.md
index 6ebfbc7b8d..2f8575f650 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/haystack.md
+++ b/apps/opik-documentation/documentation/docs/tracing/integrations/haystack.md
@@ -19,7 +19,7 @@ pip install opik haystack-ai
In addition, you can configure Opik using the `opik configure` command which will prompt you for the correct local server address or if you are using the Cloud platfrom your API key:
-```bash
+```bash pytest_codeblocks_skip="true"
opik configure
```
@@ -39,7 +39,6 @@ from haystack.dataclasses import ChatMessage
from opik.integrations.haystack import OpikConnector
-
pipe = Pipeline()
# Add the OpikConnector component to the pipeline
@@ -91,10 +90,18 @@ By default the `OpikConnector` will flush the trace to the Opik platform after e
In order to make sure that all traces are logged to the Opik platform before you exit a script, you can use the `flush` method:
```python
+from opik.integrations.haystack import OpikConnector
from haystack.tracing import tracer
+from haystack import Pipeline
-# Pipeline definition
+pipe = Pipeline()
+# Add the OpikConnector component to the pipeline
+pipe.add_component(
+ "tracer", OpikConnector("Chat example")
+)
+
+# Pipeline definition
tracer.actual_tracer.flush()
```
@@ -108,7 +115,7 @@ Disabling this feature may result in data loss if the program crashes before the
The `OpikConnector` returns the logged trace ID in the pipeline run response. You can use this ID to update the trace with feedback scores or other metadata:
-```python
+```python pytest_codeblocks_skip="true"
import opik
response = pipe.run(
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/langgraph.md b/apps/opik-documentation/documentation/docs/tracing/integrations/langgraph.md
index fde8fa5fa9..edd7d4cbd4 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/langgraph.md
+++ b/apps/opik-documentation/documentation/docs/tracing/integrations/langgraph.md
@@ -19,7 +19,7 @@ Opik provides a seamless integration with LangGraph, allowing you to easily log
To use the [`OpikTracer`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/langchain/OpikTracer.html) with LangGraph, you'll need to have both the `opik` and `langgraph` packages installed. You can install them using pip:
```bash
-pip install opik langgraph
+pip install opik langgraph langchain
```
In addition, you can configure Opik using the `opik configure` command which will prompt you for the correct local server address or if you are using the Cloud platfrom your API key:
@@ -33,21 +33,37 @@ opik configure
You can use the [`OpikTracer`](https://www.comet.com/docs/opik/python-sdk-reference/integrations/langchain/OpikTracer.html) callback with any LangGraph graph by passing it in as an argument to the `stream` or `invoke` functions:
```python
+from typing import List, Annotated
+from pydantic import BaseModel
from opik.integrations.langchain import OpikTracer
+from langchain_core.messages import HumanMessage
+from langgraph.graph import StateGraph, START, END
+from langgraph.graph.message import add_messages
# create your LangGraph graph
-graph = ...
-app = graph.compile(...)
+class State(BaseModel):
+ messages: Annotated[list, add_messages]
+def chatbot(state):
+ # Typically your LLM calls would be done here
+ return {"messages": "Hello, how can I help you today?"}
+
+graph = StateGraph(State)
+graph.add_node("chatbot", chatbot)
+graph.add_edge(START, "chatbot")
+graph.add_edge("chatbot", END)
+app = graph.compile()
+
+# Create the OpikTracer
opik_tracer = OpikTracer(graph=app.get_graph(xray=True))
# Pass the OpikTracer callback to the Graph.stream function
-for s in app.stream({"messages": [HumanMessage(content = QUESTION)]},
+for s in app.stream({"messages": [HumanMessage(content = "How to use LangGraph ?")]},
config={"callbacks": [opik_tracer]}):
print(s)
# Pass the OpikTracer callback to the Graph.invoke function
-result = app.invoke({"messages": [HumanMessage(content = QUESTION)]},
+result = app.invoke({"messages": [HumanMessage(content = "How to use LangGraph ?")]},
config={"callbacks": [opik_tracer]})
```
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/llama_index.md b/apps/opik-documentation/documentation/docs/tracing/integrations/llama_index.md
index 64ec4ce819..0a561e20b5 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/llama_index.md
+++ b/apps/opik-documentation/documentation/docs/tracing/integrations/llama_index.md
@@ -101,11 +101,7 @@ from llama_index.core import VectorStoreIndex, SimpleDirectoryReader
documents = SimpleDirectoryReader("./data/paul_graham").load_data()
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine()
-```
-
-Now that the query engine is set up, we can use it to query the data:
-```python
response = query_engine.query("What did the author do growing up?")
print(response)
```
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/ollama.md b/apps/opik-documentation/documentation/docs/tracing/integrations/ollama.md
index c4b9574be8..af773935fc 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/ollama.md
+++ b/apps/opik-documentation/documentation/docs/tracing/integrations/ollama.md
@@ -123,7 +123,7 @@ The local LLM call is now traced and logged to Opik.
In order to trace Ollama calls made with LangChain, you will need to first install the `langchain-ollama` package:
```bash
-pip install --quiet --upgrade langchain-ollama
+pip install --quiet --upgrade langchain-ollama langchain
```
You will now be able to use the `OpikTracer` class to log all your Ollama calls made with LangChain to Opik:
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/overview.md b/apps/opik-documentation/documentation/docs/tracing/integrations/overview.md
index 8b97a59037..8e8c30944c 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/overview.md
+++ b/apps/opik-documentation/documentation/docs/tracing/integrations/overview.md
@@ -16,12 +16,13 @@ Opik aims to make it as easy as possible to log, view and evaluate your LLM trac
| aisuite | Log traces for all aisuite LLM calls | [Documentation](/tracing/integrations/aisuite.md) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/aisuite.ipynb) |
| Anthropic | Log traces for all Anthropic LLM calls | [Documentation](/tracing/integrations/anthropic.md) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/anthropic.ipynb) |
| Bedrock | Log traces for all AWS Bedrock LLM calls | [Documentation](/tracing/integrations/bedrock.md) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/bedrock.ipynb) |
+| Dify | Log traces and LLM calls for your Dify Apps | [Documentation](/tracing/integrations/dify.mdx) | |
+| DSPy | Log traces for all DSPy runs | [Documentation](/tracing/integrations/dspy.md) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/dspy.ipynb) |
| LangGraph | Log traces for all LangGraph executions | [Documentation](/tracing/integrations/langgraph.md) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/langgraph.ipynb) |
| LlamaIndex | Log traces for all LlamaIndex LLM calls | [Documentation](/tracing/integrations/llama_index.md) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/llama-index.ipynb) |
| Ollama | Log traces for all Ollama LLM calls | [Documentation](/tracing/integrations/ollama.md) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/ollama.ipynb) |
| Predibase | Fine-tune and serve open-source LLMs | [Documentation](/tracing/integrations/predibase.md) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/predibase.ipynb) |
| Ragas | Evaluation framework for your Retrieval Augmented Generation (RAG) pipelines | [Documentation](/tracing/integrations/ragas.md) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/ragas.ipynb) |
| watsonx | Log traces for all watsonx LLM calls | [Documentation](/tracing/integrations/watsonx.md) | [](https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/watsonx.ipynb) |
-| Dify | Log traces and LLM calls for your Dify Apps | [Documentation](/tracing/integrations/dify.mdx) | |
If you would like to see more integrations, please open an issue on our [GitHub repository](https://github.com/comet-ml/opik/issues/new/choose).
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/predibase.md b/apps/opik-documentation/documentation/docs/tracing/integrations/predibase.md
index bd97df52ae..9575225035 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/predibase.md
+++ b/apps/opik-documentation/documentation/docs/tracing/integrations/predibase.md
@@ -1,6 +1,7 @@
---
sidebar_label: Predibase
description: Describes how to track Predibase LLM calls using Opik
+test_code_snippets: false
---
# Using Opik with Predibase
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/ragas.md b/apps/opik-documentation/documentation/docs/tracing/integrations/ragas.md
index 48fd243a44..4003e01abf 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/ragas.md
+++ b/apps/opik-documentation/documentation/docs/tracing/integrations/ragas.md
@@ -1,6 +1,7 @@
---
sidebar_label: Ragas
description: Describes how to log Ragas scores to the Opik platform
+test_code_snippets: false
---
# Ragas
diff --git a/apps/opik-documentation/documentation/docs/tracing/integrations/watsonx.md b/apps/opik-documentation/documentation/docs/tracing/integrations/watsonx.md
index 2a632cbd1b..8f61507968 100644
--- a/apps/opik-documentation/documentation/docs/tracing/integrations/watsonx.md
+++ b/apps/opik-documentation/documentation/docs/tracing/integrations/watsonx.md
@@ -1,6 +1,7 @@
---
sidebar_label: watsonx
description: Describes how to track watsonx LLM calls using Opik
+test_code_snippets: false
---
# watsonx
diff --git a/apps/opik-documentation/documentation/docs/tracing/log_distributed_traces.md b/apps/opik-documentation/documentation/docs/tracing/log_distributed_traces.md
index 5003e3cb12..7adda8ddf0 100644
--- a/apps/opik-documentation/documentation/docs/tracing/log_distributed_traces.md
+++ b/apps/opik-documentation/documentation/docs/tracing/log_distributed_traces.md
@@ -1,6 +1,7 @@
---
sidebar_label: Log Distributed Traces
description: Describes how to log distributed traces to the Opik platform
+test_code_snippets: false
---
# Log Distributed Traces
diff --git a/apps/opik-documentation/documentation/docs/tracing/sdk_configuration.mdx b/apps/opik-documentation/documentation/docs/tracing/sdk_configuration.mdx
index 312268f4ef..8609cfe786 100644
--- a/apps/opik-documentation/documentation/docs/tracing/sdk_configuration.mdx
+++ b/apps/opik-documentation/documentation/docs/tracing/sdk_configuration.mdx
@@ -32,7 +32,7 @@ opik configure
If you are self-hosting the platform, you can configure the SDK by running:
-```python
+```python pytest_codeblocks_skip="true"
import opik
opik.configure(use_local=True)
@@ -40,7 +40,7 @@ opik.configure(use_local=True)
or from the Command line:
-```bash
+```bash pytest_codeblocks_skip="true"
opik configure --use_local
```
diff --git a/apps/opik-documentation/documentation/pytest.ini b/apps/opik-documentation/documentation/pytest.ini
new file mode 100644
index 0000000000..5dd43de1b8
--- /dev/null
+++ b/apps/opik-documentation/documentation/pytest.ini
@@ -0,0 +1,7 @@
+[pytest]
+testpaths = docs
+python_files = *.md *.mdx
+log_cli = true
+log_cli_level = INFO
+asyncio_default_fixture_loop_scope = module
+norecursedirs = docs/cookbook
diff --git a/apps/opik-documentation/documentation/pytest_codeblocks/README.md b/apps/opik-documentation/documentation/pytest_codeblocks/README.md
new file mode 100644
index 0000000000..b86b4f449d
--- /dev/null
+++ b/apps/opik-documentation/documentation/pytest_codeblocks/README.md
@@ -0,0 +1,48 @@
+# Pytest Codeblocks
+
+The pytest-codeblocks extension allows you to run code blocks from a markdown file using pytest.
+
+This extension was created to ensure that:
+
+1. Each markdown file is tested in isolation using a fresh Python environment - This allows us to detect missing package install statements.
+2. It is possible to skip testing certain code blocks.
+
+## How does it work
+
+The pytest-codeblocks extension uses pytest to run the code blocks in a markdown file.
+
+The extension works by:
+
+1. Collecting all the code blocks in a markdown file
+2. Creating a new venv for each code block. Currently only Python and Bash code blocks are supported
+3. Running each code block in this new env
+
+**Note: This extension only tests that the code successfull runs without raising an error, it does not test the output of the code.**
+
+## Using the pytest-codeblocks
+
+To use the pytest-codeblocks extension, simply run:
+
+```bash
+cd apps/opik-documentation/documentation/
+
+pytest
+```
+
+## Advanced usage
+
+The `pytest-codeblocks` extension supports the following features:
+
+- Skipping all code blocks in a file: By setting `pytest_codeblocks_skip: true` in the frontmatter of the markdown file, all code blocks in the file will be skipped.
+
+- Skipping specific code blocks: Setting `pytest_codeblocks_skip="true"` in the header of the code block will skip it:
+
+ ````
+ ```python pytest_codeblocks_skip="true"
+
+ print("test")
+
+ ```
+ ````
+
+- Executing all previous code blocks: By setting `pytest_codeblocks_execute_previous: true` in the frontmatter of the markdown file, all code blocks in the file will be executed before the current code block. This is useful if you have multiple code blocks that depend on variables defined in a previous code block for example.
diff --git a/apps/opik-documentation/documentation/pytest_codeblocks/__init__.py b/apps/opik-documentation/documentation/pytest_codeblocks/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
diff --git a/apps/opik-documentation/documentation/pytest_codeblocks/environment.py b/apps/opik-documentation/documentation/pytest_codeblocks/environment.py
new file mode 100644
index 0000000000..53c1404f16
--- /dev/null
+++ b/apps/opik-documentation/documentation/pytest_codeblocks/environment.py
@@ -0,0 +1,36 @@
+from typing import Optional, List
+import os
+import tempfile
+import subprocess
+import venv
+from datetime import datetime
+
+
+def setup_env(packages: Optional[List[str]] = None):
+ """
+ Create a virtual environment and install required packages.
+
+ Args:
+ packages: Optional list of packages to install in the environment
+ """
+ # Create a virtual environment
+ env_path = os.path.join(
+ tempfile.gettempdir(), f"venv_{datetime.now().strftime('%Y%m%d%H%M%S')}"
+ )
+ venv.create(env_path, with_pip=True, clear=True)
+
+ # Get paths to executables
+ if os.name == "nt": # Windows
+ python_path = os.path.join(env_path, "Scripts", "python.exe")
+ pip_path = os.path.join(env_path, "Scripts", "pip.exe")
+ else: # Unix-like
+ python_path = os.path.join(env_path, "bin", "python")
+ pip_path = os.path.join(env_path, "bin", "pip")
+
+ # Install required packages
+ if packages:
+ subprocess.run(
+ [pip_path, "install"] + packages, capture_output=True, check=True
+ )
+
+ return env_path, python_path, pip_path
diff --git a/apps/opik-documentation/documentation/pytest_codeblocks/evaluators/__init__.py b/apps/opik-documentation/documentation/pytest_codeblocks/evaluators/__init__.py
new file mode 100644
index 0000000000..dcd9188288
--- /dev/null
+++ b/apps/opik-documentation/documentation/pytest_codeblocks/evaluators/__init__.py
@@ -0,0 +1,7 @@
+from .bash_evaluator import BashEvaluator
+from .python_evaluator import PythonEvaluator
+
+__all__ = [
+ "BashEvaluator",
+ "PythonEvaluator",
+]
diff --git a/apps/opik-documentation/documentation/pytest_codeblocks/evaluators/bash_evaluator.py b/apps/opik-documentation/documentation/pytest_codeblocks/evaluators/bash_evaluator.py
new file mode 100644
index 0000000000..d935d9053e
--- /dev/null
+++ b/apps/opik-documentation/documentation/pytest_codeblocks/evaluators/bash_evaluator.py
@@ -0,0 +1,27 @@
+import os
+import subprocess
+
+
+class BashEvaluator:
+ def __init__(self, code, file=None, start_line=None, test=True):
+ self.code = code
+ self.file = file
+ self.start_line = start_line
+ self.test = test
+
+ def set_env(self, env_path: str, python_path: str, pip_path: str):
+ self.env_path = env_path
+ self.python_path = python_path
+ self.pip_path = pip_path
+
+ def evaluate(self):
+ env = os.environ.copy()
+ env.update(
+ {
+ "PATH": f"{os.path.dirname(self.pip_path)}:{env.get('PATH', '')}",
+ "VIRTUAL_ENV": self.env_path,
+ "PYTHONPATH": os.path.dirname(self.python_path),
+ }
+ )
+
+ subprocess.run(self.code, shell=True, env=env)
diff --git a/apps/opik-documentation/documentation/pytest_codeblocks/evaluators/python_evaluator.py b/apps/opik-documentation/documentation/pytest_codeblocks/evaluators/python_evaluator.py
new file mode 100644
index 0000000000..0f2c69e800
--- /dev/null
+++ b/apps/opik-documentation/documentation/pytest_codeblocks/evaluators/python_evaluator.py
@@ -0,0 +1,39 @@
+import os
+import subprocess
+import tempfile
+
+
+class PythonEvaluator:
+ def __init__(self, code, start_line=None, history=None):
+ self.code = code
+ self.start_line = start_line
+ self.history = history or []
+
+ def set_env(self, env_path: str, python_path: str, pip_path: str):
+ self.env_path = env_path
+ self.python_path = python_path
+ self.pip_path = pip_path
+
+ def evaluate(self):
+ # Run the code in a subprocess
+ with tempfile.TemporaryDirectory() as temp_dir:
+ script_path = os.path.join(temp_dir, "script.py")
+ with open(script_path, "w") as f:
+ f.write("\n".join([*self.history, self.code]))
+
+ env = os.environ.copy()
+ env.update(
+ {
+ "PATH": f"{os.path.dirname(self.pip_path)}:{env.get('PATH', '')}",
+ "VIRTUAL_ENV": self.env_path,
+ "PYTHONPATH": os.path.dirname(self.python_path),
+ }
+ )
+
+ subprocess.run(
+ [self.python_path, script_path],
+ capture_output=True,
+ text=True,
+ env=env,
+ check=True,
+ )
diff --git a/apps/opik-documentation/documentation/pytest_codeblocks/parsing_utils.py b/apps/opik-documentation/documentation/pytest_codeblocks/parsing_utils.py
new file mode 100644
index 0000000000..48effdc887
--- /dev/null
+++ b/apps/opik-documentation/documentation/pytest_codeblocks/parsing_utils.py
@@ -0,0 +1,114 @@
+from mrkdwn_analysis import MarkdownAnalyzer
+import logging
+from . import evaluators
+from typing import List, Union
+
+LOGGER = logging.getLogger(__name__)
+
+
+def _get_code_block_language(language: str):
+ """
+ This method extracts the language of the code block based on the string that is
+ after ``` in each code block.
+ """
+ params = language.split(" ")
+ if (len(params) == 1) and params[0] == "":
+ return None
+ else:
+ return params[0]
+
+
+def _reindent_code_block(code_block):
+ first_line = code_block.split("\n")[0]
+ leading_spaces = len(first_line) - len(first_line.lstrip())
+
+ return "\n".join([x[leading_spaces:] for x in code_block.split("\n")])
+
+
+def get_page_frontmatter(path):
+ headers = {}
+ with open(path, "r") as f:
+ lines = f.read().split("\n")
+ if lines[0] != "---":
+ return headers
+ else:
+ for line in lines[1:]:
+ if line.startswith("---"):
+ break
+ if ":" in line:
+ k, v = line.split(":")
+ v = v.strip()
+ k = k.strip()
+ if v == "true" or v == "True":
+ v = True
+ if v == "false" or v == "False":
+ v = False
+ headers[k] = v
+ return headers
+
+
+def check_skip_code_block(mk_language):
+ language_params = mk_language.split(" ")
+ for params in language_params:
+ if "=" in params:
+ title, value = params.split("=")
+ if title == "pytest_codeblocks_skip" and (
+ value.strip() == "true" or value.strip() == "True"
+ ):
+ return True
+
+ return False
+
+
+def check_skip_frontmatter(path):
+ frontmatter = get_page_frontmatter(path)
+ return frontmatter.get("pytest_codeblocks_skip", False)
+
+
+def get_code_blocs(
+ path: str,
+) -> List[Union[evaluators.PythonEvaluator, evaluators.BashEvaluator]]:
+ LOGGER.debug(f"Finding code blocks in {path}")
+
+ if check_skip_frontmatter(path):
+ LOGGER.debug(f"Skipping {path} because test_code_snippets is set to false")
+ return []
+
+ page_frontmatter = get_page_frontmatter(path)
+ code_blocks = []
+ markdown = MarkdownAnalyzer(path)
+ mrkdwn_analysis_code_blocks = markdown.identify_code_blocks().get("Code block", [])
+ for i, mk_code_block in enumerate(mrkdwn_analysis_code_blocks):
+ language = _get_code_block_language(mk_code_block["language"])
+ start_line = mk_code_block["start_line"]
+
+ if language not in ["bash", "python"]:
+ LOGGER.debug(
+ f"Skipping code block in {path}:{start_line} because language '{language}' is not supported."
+ )
+ continue
+
+ if check_skip_code_block(mk_code_block["language"]):
+ LOGGER.debug(
+ f"Skipping code block in {path}:{start_line} because test is set to false."
+ )
+ continue
+
+ code_str = _reindent_code_block(mk_code_block["content"])
+ if language == "python":
+ if page_frontmatter.get("pytest_codeblocks_execute_previous", False):
+ history = [x["content"] for x in mrkdwn_analysis_code_blocks[:i]]
+ else:
+ history = []
+
+ code_blocks.append(
+ evaluators.PythonEvaluator(code_str, start_line, history=history)
+ )
+ elif language == "bash":
+ code_blocks.append(
+ evaluators.BashEvaluator(code=code_str, start_line=start_line)
+ )
+
+ LOGGER.debug(f"Found {len(code_blocks)} code blocks to test in {path}")
+
+ return code_blocks
diff --git a/apps/opik-documentation/documentation/pytest_codeblocks/pytest_integration.py b/apps/opik-documentation/documentation/pytest_codeblocks/pytest_integration.py
new file mode 100644
index 0000000000..3de19c177f
--- /dev/null
+++ b/apps/opik-documentation/documentation/pytest_codeblocks/pytest_integration.py
@@ -0,0 +1,94 @@
+import pytest
+import logging
+from pytest import Collector, ExceptionInfo, Module, Session
+from _pytest._code.code import TerminalRepr
+from typing import Union, Optional, Iterator, Any
+from pathlib import Path
+import os
+from . import evaluators, reporting, parsing_utils, environment
+
+LOGGER = logging.getLogger(__name__)
+
+
+class OpikDocsTestFile(pytest.File):
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.venv_path = None
+ self.venv_python = None
+ self.venv_pip = None
+
+ @classmethod
+ def from_parent(
+ cls, parent: Session, path: Path, **kwargs: Any
+ ) -> "OpikDocsTestFile":
+ item = super().from_parent(parent=parent, path=path, **kwargs)
+
+ return item
+
+ def collect(self) -> Iterator["OpikDocsTestItem"]:
+ code_blocks = parsing_utils.get_code_blocs(self.path)
+
+ for code_block in code_blocks:
+ if not self.venv_path:
+ LOGGER.info("Setting up venv for code snippets in:", self.path)
+ self.venv_path, self.venv_python, self.venv_pip = environment.setup_env(
+ ["opik"]
+ )
+
+ code_block.set_env(
+ env_path=self.venv_path,
+ python_path=self.venv_python,
+ pip_path=self.venv_pip,
+ )
+
+ yield OpikDocsTestItem.from_parent(
+ parent=self,
+ name=f"code_block_starting_line_{code_block.start_line}",
+ test_case=code_block,
+ )
+
+
+class OpikDocsTestItem(pytest.Item):
+ def __init__(
+ self,
+ name,
+ parent,
+ test_case: Optional[
+ Union[evaluators.PythonEvaluator, evaluators.BashEvaluator]
+ ] = None,
+ ):
+ super().__init__(name, parent)
+ self.test_case = test_case
+
+ @classmethod
+ def from_parent(cls, parent, name, test_case, **kwargs):
+ item = super().from_parent(parent=parent, name=name, **kwargs)
+ item.test_case = test_case
+ return item
+
+ def runtest(self):
+ if self.test_case is not None:
+ self.test_case.evaluate()
+
+ def repr_failure(
+ self, excinfo: ExceptionInfo[BaseException]
+ ) -> Union[str, TerminalRepr]:
+ return reporting.format_error(self.fspath, self.test_case, excinfo)
+
+
+def pytest_collect_file(parent: Collector, file_path: Path) -> Optional[Module]:
+ """Hook to collect MDX files for testing"""
+ test_path = parent.config.args[0] if parent.config.args else None
+
+ if test_path:
+ # Convert both paths to absolute and normalized form for comparison
+ test_path = os.path.abspath(os.path.normpath(test_path))
+ current_path = os.path.abspath(os.path.normpath(str(file_path)))
+
+ # Only collect if this is the specific file being tested
+ if current_path.startswith(test_path) and file_path.suffix in (".mdx", ".md"):
+ return OpikDocsTestFile.from_parent(parent=parent, path=file_path)
+ elif file_path.suffix in (".mdx", ".md"):
+ # Fallback to old behavior if no specific path provided
+ return OpikDocsTestFile.from_parent(parent=parent, path=file_path)
+ return None
diff --git a/apps/opik-documentation/documentation/pytest_codeblocks/reporting.py b/apps/opik-documentation/documentation/pytest_codeblocks/reporting.py
new file mode 100644
index 0000000000..15fc6862bb
--- /dev/null
+++ b/apps/opik-documentation/documentation/pytest_codeblocks/reporting.py
@@ -0,0 +1,36 @@
+from pytest import ExceptionInfo
+from typing import Union
+from . import evaluators
+
+
+def format_error(
+ path: str,
+ test_case: Union[evaluators.PythonEvaluator, evaluators.BashEvaluator],
+ excinfo: ExceptionInfo[BaseException],
+) -> str:
+ # Get the error type and message
+ error_type = excinfo.type.__name__
+
+ # Get the line number from the traceback
+ traceback = excinfo.traceback
+ last_entry = traceback[-1] # Get the last frame in the traceback
+ error_type = excinfo.type.__name__
+ if error_type == "CalledProcessError":
+ error = excinfo.value
+ actual_line = 0
+
+ error_msg = ""
+ skip = True
+ for line in error.stderr.split("\n"):
+ if 'script.py", line' in line:
+ skip = False
+ line_part = line.split("line")[1].split(",")[0].strip()
+ actual_line = test_case.start_line + int(line_part) - 1
+ if not skip:
+ error_msg += "\n" + line
+ else:
+ actual_line = test_case.start_line + last_entry.lineno
+ error_msg = str(excinfo.value)
+
+ # Format like a standard Python error
+ return f"{path}:{actual_line}: {error_msg}"
diff --git a/apps/opik-documentation/documentation/requirements.txt b/apps/opik-documentation/documentation/requirements.txt
index 87906ccb40..b920daf862 100644
--- a/apps/opik-documentation/documentation/requirements.txt
+++ b/apps/opik-documentation/documentation/requirements.txt
@@ -1 +1,4 @@
-jupyter
\ No newline at end of file
+jupyter
+markdown-analysis
+pytest-asyncio
+pytest-custom_exit_code
diff --git a/apps/opik-documentation/documentation/sidebars.ts b/apps/opik-documentation/documentation/sidebars.ts
index 4c42861003..7aa40cbcf1 100644
--- a/apps/opik-documentation/documentation/sidebars.ts
+++ b/apps/opik-documentation/documentation/sidebars.ts
@@ -42,7 +42,6 @@ const sidebars: SidebarsConfig = {
"tracing/log_distributed_traces",
"tracing/annotate_traces",
"tracing/cost_tracking",
- "tracing/production_monitoring",
"tracing/sdk_configuration",
"tracing/export_data",
{
@@ -58,6 +57,7 @@ const sidebars: SidebarsConfig = {
"tracing/integrations/anthropic",
"tracing/integrations/bedrock",
"tracing/integrations/dify",
+ "tracing/integrations/dspy",
"tracing/integrations/gemini",
"tracing/integrations/groq",
"tracing/integrations/haystack",
@@ -116,7 +116,7 @@ const sidebars: SidebarsConfig = {
type: "category",
label: "Production",
collapsed: true,
- items: ["production/gateway"],
+ items: ["production/gateway", "production/production_monitoring"],
},
{
type: "category",
@@ -130,6 +130,7 @@ const sidebars: SidebarsConfig = {
"cookbook/aisuite",
"cookbook/anthropic",
"cookbook/bedrock",
+ "cookbook/dspy",
"cookbook/gemini",
"cookbook/groq",
"cookbook/haystack",
diff --git a/apps/opik-documentation/documentation/static/img/cookbook/dspy_trace_cookbook.png b/apps/opik-documentation/documentation/static/img/cookbook/dspy_trace_cookbook.png
new file mode 100644
index 0000000000..97c966f34e
Binary files /dev/null and b/apps/opik-documentation/documentation/static/img/cookbook/dspy_trace_cookbook.png differ
diff --git a/apps/opik-documentation/python-sdk-docs/requirements.txt b/apps/opik-documentation/python-sdk-docs/requirements.txt
index 525ad61969..33b9883f2d 100644
--- a/apps/opik-documentation/python-sdk-docs/requirements.txt
+++ b/apps/opik-documentation/python-sdk-docs/requirements.txt
@@ -10,3 +10,5 @@ langchain_core
botocore
anthropic
haystack-ai
+dspy-ai
+cloudpickle
diff --git a/apps/opik-documentation/python-sdk-docs/source/index.rst b/apps/opik-documentation/python-sdk-docs/source/index.rst
index 9c07d8d9ad..c9dbcc4a7d 100644
--- a/apps/opik-documentation/python-sdk-docs/source/index.rst
+++ b/apps/opik-documentation/python-sdk-docs/source/index.rst
@@ -165,12 +165,13 @@ You can learn more about the `opik` python SDK in the following sections:
:caption: Integrations
:maxdepth: 1
- integrations/openai/index
integrations/anthropic/index
- integrations/langchain/index
- integrations/haystack/index
integrations/bedrock/index
+ integrations/dspy/index
+ integrations/haystack/index
+ integrations/langchain/index
integrations/llama_index/index
+ integrations/openai/index
.. toctree::
:caption: Evaluation
diff --git a/apps/opik-documentation/python-sdk-docs/source/integrations/dspy/OpikCallback.rst b/apps/opik-documentation/python-sdk-docs/source/integrations/dspy/OpikCallback.rst
new file mode 100644
index 0000000000..f758737690
--- /dev/null
+++ b/apps/opik-documentation/python-sdk-docs/source/integrations/dspy/OpikCallback.rst
@@ -0,0 +1,5 @@
+OpikCallback
+============
+
+.. autoclass:: opik.integrations.dspy.OpikCallback
+ :members:
diff --git a/apps/opik-documentation/python-sdk-docs/source/integrations/dspy/index.rst b/apps/opik-documentation/python-sdk-docs/source/integrations/dspy/index.rst
new file mode 100644
index 0000000000..ece283efbd
--- /dev/null
+++ b/apps/opik-documentation/python-sdk-docs/source/integrations/dspy/index.rst
@@ -0,0 +1,32 @@
+DSPy
+====
+
+Opik integrates with DSPy to allow you to log your DSPy runs to the Opik platform::
+
+ import dspy
+ from opik.integrations.dspy.callback import OpikCallback
+
+ project_name = "DSPY"
+
+ lm = dspy.LM(
+ model="openai/gpt-4o-mini",
+ )
+ dspy.configure(lm=lm)
+
+
+ opik_callback = OpikCallback(project_name=project_name)
+ dspy.settings.configure(
+ callbacks=[opik_callback],
+ )
+
+ cot = dspy.ChainOfThought("question -> answer")
+ cot(question="What is the meaning of life?")
+
+
+You can learn more about the `OpikCallback` in the following section:
+
+.. toctree::
+ :maxdepth: 4
+ :titlesonly:
+
+ OpikCallback
diff --git a/apps/opik-frontend/src/api/datasets/useDatasetCreateMutation.ts b/apps/opik-frontend/src/api/datasets/useDatasetCreateMutation.ts
index 165d1399a7..74c4649cf1 100644
--- a/apps/opik-frontend/src/api/datasets/useDatasetCreateMutation.ts
+++ b/apps/opik-frontend/src/api/datasets/useDatasetCreateMutation.ts
@@ -1,11 +1,10 @@
import { useMutation, useQueryClient } from "@tanstack/react-query";
import { AxiosError } from "axios";
import get from "lodash/get";
-import last from "lodash/last";
-
import api, { DATASETS_REST_ENDPOINT } from "@/api/api";
import { Dataset } from "@/types/datasets";
import { useToast } from "@/components/ui/use-toast";
+import { extractIdFromLocation } from "@/lib/utils";
type UseDatasetCreateMutationParams = {
dataset: Partial
+
+