diff --git a/404.html b/404.html index daba0179..41f3365b 100644 --- a/404.html +++ b/404.html @@ -670,6 +670,32 @@

404 - Not found

Material for MkDocs + + +
+ + + + + + + + + + + + + + + + + + + + + + +
diff --git a/asyncio/index.html b/asyncio/index.html index 5509179b..739e9115 100644 --- a/asyncio/index.html +++ b/asyncio/index.html @@ -808,6 +808,32 @@

Asyncio + + + + + + + + + + + + + + + + + + + + + + + diff --git a/chat-prompting/index.html b/chat-prompting/index.html index 57d88050..2069e333 100644 --- a/chat-prompting/index.html +++ b/chat-prompting/index.html @@ -1095,6 +1095,32 @@

AnyMessage + + + + + + + + + + + + + + + + + + + + + + + diff --git a/configuration/index.html b/configuration/index.html index 52b57ac2..b1d989ce 100644 --- a/configuration/index.html +++ b/configuration/index.html @@ -485,6 +485,17 @@ + + @@ -495,6 +506,111 @@ + + + + + + @@ -669,6 +785,98 @@ + + + @@ -691,33 +899,66 @@

LLM Configuration

-

Magentic supports multiple "backends" (LLM providers). These are

- -

The backend and LLM (ChatModel) used by magentic can be configured in several ways. When a magentic function is called, the ChatModel to use follows this order of preference

+

Other OpenAI-compatible APIs

+

When using the openai backend, setting the MAGENTIC_OPENAI_BASE_URL environment variable or using OpenaiChatModel(..., base_url="http://localhost:8080") in code allows you to use magentic with any OpenAI-compatible API e.g. Azure OpenAI Service, LiteLLM OpenAI Proxy Server, LocalAI. Note that if the API does not support tool calls then you will not be able to create prompt-functions that return Python objects, but other features of magentic will still work.

+

To use Azure with the openai backend you will need to set the MAGENTIC_OPENAI_API_TYPE environment variable to "azure" or use OpenaiChatModel(..., api_type="azure"), and also set the environment variables needed by the openai package to access Azure. See https://github.com/openai/openai-python#microsoft-azure-openai

+

Anthropic

+

This uses the anthropic Python package and supports all features of magentic.

+

Install the magentic package with the anthropic extra, or install the anthropic package directly.

+
pip install "magentic[anthropic]"
+
+

Then import the AnthropicChatModel class.

+
from magentic.chat_model.anthropic_chat_model import AnthropicChatModel
+
+model = AnthropicChatModel("claude-3-5-sonnet-latest")
+
+

LiteLLM

+

This uses the litellm Python package to enable querying LLMs from many different providers. Note: some models may not support all features of magentic e.g. function calling/structured output and streaming.

+

Install the magentic package with the litellm extra, or install the litellm package directly.

+
pip install "magentic[litellm]"
+
+

Then import the LitellmChatModel class.

+
from magentic.chat_model.litellm_chat_model import LitellmChatModel
+
+model = LitellmChatModel("gpt-4o")
+
+

Mistral

+

This uses the openai Python package with some small modifications to make the API queries compatible with the Mistral API. It supports all features of magentic. However tool calls (including structured outputs) are not streamed so are received all at once.

+

Note: a future version of magentic might switch to using the mistral Python package.

+

No additional installation is required. Just import the MistralChatModel class.

+
from magentic.chat_model.mistral_chat_model import MistralChatModel
+
+model = MistralChatModel("mistral-large-latest")
+
+

Configure a Backend

+

The default ChatModel used by magentic (in @prompt, @chatprompt, etc.) can be configured in several ways. When a prompt-function or chatprompt-function is called, the ChatModel to use follows this order of preference

  1. The ChatModel instance provided as the model argument to the magentic decorator
  2. The current chat model context, created using with MyChatModel:
    3. -
  3. The global ChatModel created from environment variables and the default settings in src/magentic/settings.py
    4. +
  4. The global ChatModel created from environment variables and the default settings in src/magentic/settings.py
+

The following code snippet demonstrates this behavior:

from magentic import OpenaiChatModel, prompt
-from magentic.chat_model.litellm_chat_model import LitellmChatModel
+from magentic.chat_model.anthropic_chat_model import AnthropicChatModel
 
 
 @prompt("Say hello")
@@ -726,16 +967,16 @@ 
LLM Configuration@prompt(
     "Say hello",
-    model=LitellmChatModel("ollama_chat/llama3"),
+    model=AnthropicChatModel("claude-3-5-sonnet-latest"),
 )
-def say_hello_litellm() -> str: ...
+def say_hello_anthropic() -> str: ...
 
 
 say_hello()  # Uses env vars or default settings
 
-with OpenaiChatModel("gpt-3.5-turbo", temperature=1):
-    say_hello()  # Uses openai with gpt-3.5-turbo and temperature=1 due to context manager
-    say_hello_litellm()  # Uses litellm with ollama_chat/llama3 because explicitly configured
+with OpenaiChatModel("gpt-4o-mini", temperature=1):
+    say_hello()  # Uses openai with gpt-4o-mini and temperature=1 due to context manager
+    say_hello_anthropic()  # Uses Anthropic claude-3-5-sonnet-latest because explicitly configured

The following environment variables can be set.

@@ -864,8 +1105,6 @@

LLM Configuration in code allows you to use magentic with any OpenAI-compatible API e.g. Azure OpenAI Service, LiteLLM OpenAI Proxy Server, LocalAI. Note that if the API does not support tool calls then you will not be able to create prompt-functions that return Python objects, but other features of magentic will still work.

-

To use Azure with the openai backend you will need to set the MAGENTIC_OPENAI_API_TYPE environment variable to "azure" or use OpenaiChatModel(..., api_type="azure"), and also set the environment variables needed by the openai package to access Azure. See https://github.com/openai/openai-python#microsoft-azure-openai

@@ -942,6 +1181,32 @@

LLM Configuration + + + + + + + + + + + + + + + + + + + + + + + diff --git a/examples/chain_of_verification/index.html b/examples/chain_of_verification/index.html index 76f9e753..92ec8dc3 100644 --- a/examples/chain_of_verification/index.html +++ b/examples/chain_of_verification/index.html @@ -1551,6 +1551,32 @@

Chain of Verification (CoVe) + + + + + + + + + + + + + + + + + + + + + + + diff --git a/examples/rag_github/index.html b/examples/rag_github/index.html index 2c9306fd..e85dc9a1 100644 --- a/examples/rag_github/index.html +++ b/examples/rag_github/index.html @@ -1667,6 +1667,32 @@

Retrieval-Augmented Generati Material for MkDocs + + +
+ + + + + + + + + + + + + + + + + + + + + + +
diff --git a/examples/registering_custom_type/index.html b/examples/registering_custom_type/index.html index ac728e33..276d8d17 100644 --- a/examples/registering_custom_type/index.html +++ b/examples/registering_custom_type/index.html @@ -1636,6 +1636,32 @@

Registering a Custom Type + + + + + + + + + + + + + + + + + + + + + + + diff --git a/examples/vision_renaming_screenshots/index.html b/examples/vision_renaming_screenshots/index.html index 6f49ff6c..0aac8ee1 100644 --- a/examples/vision_renaming_screenshots/index.html +++ b/examples/vision_renaming_screenshots/index.html @@ -1589,6 +1589,32 @@

Renaming Screenshots with GPT Visi Material for MkDocs + + +
+ + + + + + + + + + + + + + + + + + + + + + +
diff --git a/formatting/index.html b/formatting/index.html index 3220add3..ff52cf4d 100644 --- a/formatting/index.html +++ b/formatting/index.html @@ -933,6 +933,32 @@

Custom Formatting + + + + + + + + + + + + + + + + + + + + + + + diff --git a/function-calling/index.html b/function-calling/index.html index f3de1319..8d4fd46f 100644 --- a/function-calling/index.html +++ b/function-calling/index.html @@ -1103,6 +1103,32 @@

ConfigDict + + + + + + + + + + + + + + + + + + + + + + + diff --git a/index.html b/index.html index e33791ec..97a2c7b2 100644 --- a/index.html +++ b/index.html @@ -867,19 +867,16 @@

Overview

-

Easily integrate Large Language Models into your Python code. Simply use the @prompt and @chatprompt decorators to create functions that return structured output from the LLM. Mix LLM queries and function calling with regular Python code to create complex logic.

+

Seamlessly integrate Large Language Models into Python code. Use the @prompt and @chatprompt decorators to create functions that return structured output from an LLM. Combine LLM queries and tool use with traditional Python code to build complex agentic systems.

Features

Installation

pip install magentic
@@ -1089,6 +1086,32 @@ 
@prompt_chain
   
+
+ +
+ + + + + + + + + + + + + + + + + + + + + + +
diff --git a/logging-and-tracing/index.html b/logging-and-tracing/index.html index e5af5862..74bc98cd 100644 --- a/logging-and-tracing/index.html +++ b/logging-and-tracing/index.html @@ -1023,6 +1023,32 @@

Enabling Debug Logging + + +
+ + + + + + + + + + + + + + + + + + + + + + +
diff --git a/retrying/index.html b/retrying/index.html index 5d329f30..5a462326 100644 --- a/retrying/index.html +++ b/retrying/index.html @@ -917,6 +917,32 @@

RetryChatModel + + + + + + + + + + + + + + + + + + + + + + + diff --git a/search/search_index.json b/search/search_index.json index 73a247dd..af38ed4a 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Overview","text":"

Easily integrate Large Language Models into your Python code. Simply use the @prompt and @chatprompt decorators to create functions that return structured output from the LLM. Mix LLM queries and function calling with regular Python code to create complex logic.

"},{"location":"#features","title":"Features","text":"
  • Structured Outputs using pydantic models and built-in python types.
  • Chat Prompting to enable few-shot prompting with structured examples.
  • Function Calling and Parallel Function Calling via the FunctionCall and ParallelFunctionCall return types.
  • Formatting to naturally insert python objects into prompts.
  • Asyncio. Simply use async def when defining a magentic function.
  • Streaming structured outputs to use them as they are being generated.
  • Vision to easily get structured outputs from images.
  • LLM-Assisted Retries to improve LLM adherence to complex output schemas.
  • Multiple LLM providers including OpenAI and Anthropic. See Configuration.
  • Type Annotations to work nicely with linters and IDEs.
"},{"location":"#installation","title":"Installation","text":"
pip install magentic\n

or using uv

uv add magentic\n

Configure your OpenAI API key by setting the OPENAI_API_KEY environment variable. To configure a different LLM provider see Configuration for more.

"},{"location":"#usage","title":"Usage","text":""},{"location":"#prompt","title":"@prompt","text":"

The @prompt decorator allows you to define a template for a Large Language Model (LLM) prompt as a Python function. When this function is called, the arguments are inserted into the template, then this prompt is sent to an LLM which generates the function output.

from magentic import prompt\n\n\n@prompt('Add more \"dude\"ness to: {phrase}')\ndef dudeify(phrase: str) -> str: ...  # No function body as this is never executed\n\n\ndudeify(\"Hello, how are you?\")\n# \"Hey, dude! What's up? How's it going, my man?\"\n

The @prompt decorator will respect the return type annotation of the decorated function. This can be any type supported by pydantic including a pydantic model.

from magentic import prompt\nfrom pydantic import BaseModel\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\n@prompt(\"Create a Superhero named {name}.\")\ndef create_superhero(name: str) -> Superhero: ...\n\n\ncreate_superhero(\"Garden Man\")\n# Superhero(name='Garden Man', age=30, power='Control over plants', enemies=['Pollution Man', 'Concrete Woman'])\n

See Structured Outputs for more.

"},{"location":"#chatprompt","title":"@chatprompt","text":"

The @chatprompt decorator works just like @prompt but allows you to pass chat messages as a template rather than a single text prompt. This can be used to provide a system message or for few-shot prompting where you provide example responses to guide the model's output. Format fields denoted by curly braces {example} will be filled in all messages (except FunctionResultMessage).

from magentic import chatprompt, AssistantMessage, SystemMessage, UserMessage\nfrom pydantic import BaseModel\n\n\nclass Quote(BaseModel):\n    quote: str\n    character: str\n\n\n@chatprompt(\n    SystemMessage(\"You are a movie buff.\"),\n    UserMessage(\"What is your favorite quote from Harry Potter?\"),\n    AssistantMessage(\n        Quote(\n            quote=\"It does not do to dwell on dreams and forget to live.\",\n            character=\"Albus Dumbledore\",\n        )\n    ),\n    UserMessage(\"What is your favorite quote from {movie}?\"),\n)\ndef get_movie_quote(movie: str) -> Quote: ...\n\n\nget_movie_quote(\"Iron Man\")\n# Quote(quote='I am Iron Man.', character='Tony Stark')\n

See Chat Prompting for more.

"},{"location":"#functioncall","title":"FunctionCall","text":"

An LLM can also decide to call functions. In this case the @prompt-decorated function returns a FunctionCall object which can be called to execute the function using the arguments provided by the LLM.

from typing import Literal\n\nfrom magentic import prompt, FunctionCall\n\n\ndef search_twitter(query: str, category: Literal[\"latest\", \"people\"]) -> str:\n    \"\"\"Searches Twitter for a query.\"\"\"\n    print(f\"Searching Twitter for {query!r} in category {category!r}\")\n    return \"<twitter results>\"\n\n\ndef search_youtube(query: str, channel: str = \"all\") -> str:\n    \"\"\"Searches YouTube for a query.\"\"\"\n    print(f\"Searching YouTube for {query!r} in channel {channel!r}\")\n    return \"<youtube results>\"\n\n\n@prompt(\n    \"Use the appropriate search function to answer: {question}\",\n    functions=[search_twitter, search_youtube],\n)\ndef perform_search(question: str) -> FunctionCall[str]: ...\n\n\noutput = perform_search(\"What is the latest news on LLMs?\")\nprint(output)\n# > FunctionCall(<function search_twitter at 0x10c367d00>, 'LLMs', 'latest')\noutput()\n# > Searching Twitter for 'Large Language Models news' in category 'latest'\n# '<twitter results>'\n

See Function Calling for more.

"},{"location":"#prompt_chain","title":"@prompt_chain","text":"

Sometimes the LLM requires making one or more function calls to generate a final answer. The @prompt_chain decorator will resolve FunctionCall objects automatically and pass the output back to the LLM to continue until the final answer is reached.

In the following example, when describe_weather is called the LLM first calls the get_current_weather function, then uses the result of this to formulate its final answer which gets returned.

from magentic import prompt_chain\n\n\ndef get_current_weather(location, unit=\"fahrenheit\"):\n    \"\"\"Get the current weather in a given location\"\"\"\n    # Pretend to query an API\n    return {\n        \"location\": location,\n        \"temperature\": \"72\",\n        \"unit\": unit,\n        \"forecast\": [\"sunny\", \"windy\"],\n    }\n\n\n@prompt_chain(\n    \"What's the weather like in {city}?\",\n    functions=[get_current_weather],\n)\ndef describe_weather(city: str) -> str: ...\n\n\ndescribe_weather(\"Boston\")\n# 'The current weather in Boston is 72\u00b0F and it is sunny and windy.'\n

LLM-powered functions created using @prompt, @chatprompt and @prompt_chain can be supplied as functions to other @prompt/@prompt_chain decorators, just like regular python functions. This enables increasingly complex LLM-powered functionality, while allowing individual components to be tested and improved in isolation.

"},{"location":"asyncio/","title":"Asyncio","text":"

Asynchronous functions / coroutines can be used to concurrently query the LLM. This can greatly increase the overall speed of generation, and also allow other asynchronous code to run while waiting on LLM output. In the below example, the LLM generates a description for each US president while it is waiting on the next one in the list. Measuring the characters generated per second shows that this example achieves a 7x speedup over serial processing.

import asyncio\nfrom time import time\nfrom typing import AsyncIterable\n\nfrom magentic import prompt\n\n\n@prompt(\"List ten presidents of the United States\")\nasync def iter_presidents() -> AsyncIterable[str]: ...\n\n\n@prompt(\"Tell me more about {topic}\")\nasync def tell_me_more_about(topic: str) -> str: ...\n\n\n# For each president listed, generate a description concurrently\nstart_time = time()\ntasks = []\nasync for president in await iter_presidents():\n    # Use asyncio.create_task to schedule the coroutine for execution before awaiting it\n    # This way descriptions will start being generated while the list of presidents is still being generated\n    task = asyncio.create_task(tell_me_more_about(president))\n    tasks.append(task)\n\ndescriptions = await asyncio.gather(*tasks)\n\n# Measure the characters per second\ntotal_chars = sum(len(desc) for desc in descriptions)\ntime_elapsed = time() - start_time\nprint(total_chars, time_elapsed, total_chars / time_elapsed)\n# 24575 28.70 856.07\n\n\n# Measure the characters per second to describe a single president\nstart_time = time()\nout = await tell_me_more_about(\"George Washington\")\ntime_elapsed = time() - start_time\nprint(len(out), time_elapsed, len(out) / time_elapsed)\n# 2206 18.72 117.78\n
"},{"location":"chat-prompting/","title":"Chat Prompting","text":""},{"location":"chat-prompting/#chatprompt","title":"@chatprompt","text":"

The @chatprompt decorator works just like @prompt but allows you to pass chat messages as a template rather than a single text prompt. This can be used to provide a system message or for few-shot prompting where you provide example responses to guide the model's output. Format fields denoted by curly braces {example} will be filled in all messages (except FunctionResultMessage).

from magentic import chatprompt, AssistantMessage, SystemMessage, UserMessage\nfrom pydantic import BaseModel\n\n\nclass Quote(BaseModel):\n    quote: str\n    character: str\n\n\n@chatprompt(\n    SystemMessage(\"You are a movie buff.\"),\n    UserMessage(\"What is your favorite quote from Harry Potter?\"),\n    AssistantMessage(\n        Quote(\n            quote=\"It does not do to dwell on dreams and forget to live.\",\n            character=\"Albus Dumbledore\",\n        )\n    ),\n    UserMessage(\"What is your favorite quote from {movie}?\"),\n)\ndef get_movie_quote(movie: str) -> Quote: ...\n\n\nget_movie_quote(\"Iron Man\")\n# Quote(quote='I am Iron Man.', character='Tony Stark')\n
"},{"location":"chat-prompting/#escape_braces","title":"escape_braces","text":"

To prevent curly braces from being interpreted as format fields, use the escape_braces function to escape them in strings.

from magentic.chatprompt import escape_braces\n\nstring_with_braces = \"Curly braces like {example} will be filled in!\"\nescaped_string = escape_braces(string_with_braces)\n# 'Curly braces {{example}} will be filled in!'\nescaped_string.format(example=\"test\")\n# 'Curly braces {example} will be filled in!'\n
"},{"location":"chat-prompting/#placeholder","title":"Placeholder","text":"

The Placeholder class enables templating of AssistantMessage content within the @chatprompt decorator. This allows dynamic changing of the messages used to prompt the model based on the arguments provided when the function is called.

from magentic import chatprompt, AssistantMessage, Placeholder, UserMessage\nfrom pydantic import BaseModel\n\n\nclass Quote(BaseModel):\n    quote: str\n    character: str\n\n\n@chatprompt(\n    UserMessage(\"Tell me a quote from {movie}\"),\n    AssistantMessage(Placeholder(Quote, \"quote\")),\n    UserMessage(\"What is a similar quote from the same movie?\"),\n)\ndef get_similar_quote(movie: str, quote: Quote) -> Quote: ...\n\n\nget_similar_quote(\n    movie=\"Star Wars\",\n    quote=Quote(quote=\"I am your father\", character=\"Darth Vader\"),\n)\n# Quote(quote='The Force will be with you, always.', character='Obi-Wan Kenobi')\n

Placeholder can also be utilized in the format method of custom Message subclasses to provide an explicit way of inserting values from the function arguments. For example, see UserImageMessage in (TODO: link to GPT-vision page).

"},{"location":"chat-prompting/#functioncall","title":"FunctionCall","text":"

The content of an AssistantMessage can be a FunctionCall. This can be used to demonstrate to the LLM when/how it should call a function.

from magentic import (\n    chatprompt,\n    AssistantMessage,\n    FunctionCall,\n    UserMessage,\n    SystemMessage,\n)\n\n\ndef change_music_volume(increment: int) -> int:\n    \"\"\"Change music volume level. Min 1, max 10.\"\"\"\n    print(f\"Music volume change: {increment}\")\n\n\ndef order_food(food: str, amount: int):\n    \"\"\"Order food.\"\"\"\n    print(f\"Ordered {amount} {food}\")\n\n\n@chatprompt(\n    SystemMessage(\n        \"You are hosting a party and must keep the guests happy.\"\n        \"Call functions as needed. Do not respond directly.\"\n    ),\n    UserMessage(\"It's pretty loud in here!\"),\n    AssistantMessage(FunctionCall(change_music_volume, -2)),\n    UserMessage(\"{request}\"),\n    functions=[change_music_volume, order_food],\n)\ndef adjust_for_guest(request: str) -> FunctionCall[None]: ...\n\n\nfunc = adjust_for_guest(\"Do you have any more food?\")\nfunc()\n# Ordered 3 pizza\n
"},{"location":"chat-prompting/#functionresultmessage","title":"FunctionResultMessage","text":"

To include the result of calling the function in the messages use a FunctionResultMessage. This takes a FunctionCall instance as its second argument. The same FunctionCall instance must be passed to an AssistantMessage and the corresponding FunctionResultMessage so that the result can be correctly linked back to the function call that created it.

from magentic import (\n    chatprompt,\n    AssistantMessage,\n    FunctionCall,\n    FunctionResultMessage,\n    UserMessage,\n)\n\n\ndef plus(a: int, b: int) -> int:\n    return a + b\n\n\nplus_1_2 = FunctionCall(plus, 1, 2)\n\n\n@chatprompt(\n    UserMessage(\"Use the plus function to add 1 and 2.\"),\n    AssistantMessage(plus_1_2),\n    FunctionResultMessage(3, plus_1_2),\n    UserMessage(\"Now add 4 to the result.\"),\n    functions=[plus],\n)\ndef do_math() -> FunctionCall[int]: ...\n\n\ndo_math()\n# FunctionCall(<function plus at 0x10a0829e0>, 3, 4)\n
"},{"location":"chat-prompting/#anymessage","title":"AnyMessage","text":"

The AnyMessage type can be used for (de)serialization of Message objects, or as a return type in prompt-functions. This allows you to create prompt-functions to do things like summarize a chat history into fewer messages, or even to create a set of messages that you can use in a chatprompt-function.

from magentic import AnyMessage, prompt\n\n\n@prompt(\"Create an example of few-shot prompting for a chatbot\")\ndef make_few_shot_prompt() -> list[AnyMessage]: ...\n\n\nmake_few_shot_prompt()\n# [SystemMessage('You are a helpful and knowledgeable assistant. You answer questions promptly and accurately. Always be polite and concise.'),\n#  UserMessage('What\u2019s the weather like today?'),\n#  AssistantMessage[Any]('The weather today is sunny with a high of 75\u00b0F (24\u00b0C) and a low of 55\u00b0F (13\u00b0C). No chance of rain.'),\n#  UserMessage('Can you explain the theory of relativity in simple terms?'),\n#  AssistantMessage[Any]('Sure! The theory of relativity, developed by Albert Einstein, has two main parts: Special Relativity and General Relativity. Special Relativity is about how time and space are linked for objects moving at a consistent speed in a straight line. It shows that time can slow down or speed up depending on how fast you are moving compared to something else. General Relativity adds gravity into the mix and shows that massive objects cause space to bend and warp, which we feel as gravity.'),\n#  UserMessage('How do I bake a chocolate cake?'),\n#  AssistantMessage[Any](\"Here's a simple recipe for a chocolate cake:\\n\\nIngredients:\\n- 1 and 3/4 cups all-purpose flour\\n- 1 and 1/2 cups granulated sugar\\n- 3/4 cup cocoa powder\\n- 1 and 1/2 teaspoons baking powder\\n- 1 and 1/2 teaspoons baking soda\\n- 1 teaspoon salt\\n- 2 large eggs\\n- 1 cup whole milk\\n- 1/2 cup vegetable oil\\n- 2 teaspoons vanilla extract\\n- 1 cup boiling water\\n\\nInstructions:\\n1. Preheat your oven to 350\u00b0F (175\u00b0C). Grease and flour two 9-inch round baking pans.\\n2. In a large bowl, whisk together the flour, sugar, cocoa powder, baking powder, baking soda, and salt.\\n3. Add the eggs, milk, oil, and vanilla. Beat on medium speed for 2 minutes.\\n4. Stir in the boiling water (batter will be thin).\\n5. Pour the batter evenly into the prepared pans.\\n6. Bake for 30 to 35 minutes or until a toothpick inserted into the center comes out clean.\\n7. Cool the cakes in the pans for 10 minutes, then remove them from the pans and cool completely on a wire rack.\\n8. Frost with your favorite chocolate frosting and enjoy!\")]\n

For (de)serialization, check out TypeAdapter from pydantic. See more on the pydantic Type Adapter docs page.

from magentic import AnyMessage\nfrom pydantic import TypeAdapter\n\n\nmessages = [\n    {\"role\": \"system\", \"content\": \"Hello\"},\n    {\"role\": \"user\", \"content\": \"Hello\"},\n    {\"role\": \"assistant\", \"content\": \"Hello\"},\n    {\"role\": \"tool\", \"content\": 3, \"tool_call_id\": \"unique_id\"},\n]\nTypeAdapter(list[AnyMessage]).validate_python(messages)\n# [SystemMessage('Hello'),\n#  UserMessage('Hello'),\n#  AssistantMessage[Any]('Hello'),\n#  ToolResultMessage[Any](3, self.tool_call_id='unique_id')]\n
"},{"location":"configuration/","title":"LLM Configuration","text":"

Magentic supports multiple \"backends\" (LLM providers). These are

  • openai : the default backend that uses the openai Python package. Supports all features of magentic. 
    from magentic import OpenaiChatModel\n
  • anthropic : uses the anthropic Python package. Supports all features of magentic, however streaming responses are currently received all at once. 
    pip install \"magentic[anthropic]\"\n
    from magentic.chat_model.anthropic_chat_model import AnthropicChatModel\n
  • litellm : uses the litellm Python package to enable querying LLMs from many different providers. Note: some models may not support all features of magentic e.g. function calling/structured output and streaming. 
    pip install \"magentic[litellm]\"\n
    from magentic.chat_model.litellm_chat_model import LitellmChatModel\n
  • mistral : uses the openai Python package with some small modifications to make the API queries compatible with the Mistral API. Supports all features of magentic, however tool calls (including structured outputs) are not streamed so are received all at once. Note: a future version of magentic might switch to using the mistral Python package. 
    from magentic.chat_model.mistral_chat_model import MistralChatModel\n

The backend and LLM (ChatModel) used by magentic can be configured in several ways. When a magentic function is called, the ChatModel to use follows this order of preference

  1. The ChatModel instance provided as the model argument to the magentic decorator
  2. The current chat model context, created using with MyChatModel:
  3. The global ChatModel created from environment variables and the default settings in src/magentic/settings.py
from magentic import OpenaiChatModel, prompt\nfrom magentic.chat_model.litellm_chat_model import LitellmChatModel\n\n\n@prompt(\"Say hello\")\ndef say_hello() -> str: ...\n\n\n@prompt(\n    \"Say hello\",\n    model=LitellmChatModel(\"ollama_chat/llama3\"),\n)\ndef say_hello_litellm() -> str: ...\n\n\nsay_hello()  # Uses env vars or default settings\n\nwith OpenaiChatModel(\"gpt-3.5-turbo\", temperature=1):\n    say_hello()  # Uses openai with gpt-3.5-turbo and temperature=1 due to context manager\n    say_hello_litellm()  # Uses litellm with ollama_chat/llama3 because explicitly configured\n

The following environment variables can be set.

Environment Variable Description Example MAGENTIC_BACKEND The package to use as the LLM backend anthropic / openai / litellm MAGENTIC_ANTHROPIC_MODEL Anthropic model claude-3-haiku-20240307 MAGENTIC_ANTHROPIC_API_KEY Anthropic API key to be used by magentic sk-... MAGENTIC_ANTHROPIC_BASE_URL Base URL for an Anthropic-compatible API http://localhost:8080 MAGENTIC_ANTHROPIC_MAX_TOKENS Max number of generated tokens 1024 MAGENTIC_ANTHROPIC_TEMPERATURE Temperature 0.5 MAGENTIC_LITELLM_MODEL LiteLLM model claude-2 MAGENTIC_LITELLM_API_BASE The base url to query http://localhost:11434 MAGENTIC_LITELLM_MAX_TOKENS LiteLLM max number of generated tokens 1024 MAGENTIC_LITELLM_TEMPERATURE LiteLLM temperature 0.5 MAGENTIC_MISTRAL_MODEL Mistral model mistral-large-latest MAGENTIC_MISTRAL_API_KEY Mistral API key to be used by magentic XEG... MAGENTIC_MISTRAL_BASE_URL Base URL for an Mistral-compatible API http://localhost:8080 MAGENTIC_MISTRAL_MAX_TOKENS Max number of generated tokens 1024 MAGENTIC_MISTRAL_SEED Seed for deterministic sampling 42 MAGENTIC_MISTRAL_TEMPERATURE Temperature 0.5 MAGENTIC_OPENAI_MODEL OpenAI model gpt-4 MAGENTIC_OPENAI_API_KEY OpenAI API key to be used by magentic sk-... MAGENTIC_OPENAI_API_TYPE Allowed options: \"openai\", \"azure\" azure MAGENTIC_OPENAI_BASE_URL Base URL for an OpenAI-compatible API http://localhost:8080 MAGENTIC_OPENAI_MAX_TOKENS OpenAI max number of generated tokens 1024 MAGENTIC_OPENAI_SEED Seed for deterministic sampling 42 MAGENTIC_OPENAI_TEMPERATURE OpenAI temperature 0.5

When using the openai backend, setting the MAGENTIC_OPENAI_BASE_URL environment variable or using OpenaiChatModel(..., base_url=\"http://localhost:8080\") in code allows you to use magentic with any OpenAI-compatible API e.g. Azure OpenAI Service, LiteLLM OpenAI Proxy Server, LocalAI. Note that if the API does not support tool calls then you will not be able to create prompt-functions that return Python objects, but other features of magentic will still work.

To use Azure with the openai backend you will need to set the MAGENTIC_OPENAI_API_TYPE environment variable to \"azure\" or use OpenaiChatModel(..., api_type=\"azure\"), and also set the environment variables needed by the openai package to access Azure. See https://github.com/openai/openai-python#microsoft-azure-openai

"},{"location":"formatting/","title":"Formatting","text":""},{"location":"formatting/#the-format-method","title":"The format Method","text":"

Functions created using magentic decorators expose a format method that accepts the same parameters as the function itself but returns the completed prompt that will be sent to the model. For @prompt this method returns a string, and for @chatprompt it returns a list of Message objects. The format method can be used to test that the final prompt created by a magentic function is formatted as expected.

from magentic import prompt\n\n\n@prompt(\"Write a short poem about {topic}.\")\ndef create_poem(topic: str) -> str: ...\n\n\ncreate_poem.format(\"fruit\")\n# 'Write a short poem about fruit.'\n
"},{"location":"formatting/#classes-for-formatting","title":"Classes for Formatting","text":"

By default, when a list is used in a prompt template string it is formatted using its Python representation.

from magentic import prompt\nfrom magentic.formatting import BulletedList\n\n\n@prompt(\"Continue the list:\\n{items}\")\ndef get_next_items(items: list[str]) -> list[str]: ...\n\n\nitems = [\"apple\", \"banana\", \"cherry\"]\nprint(get_next_items.format(items=items))\n# Continue the list:\n# ['apple', 'banana', 'cherry']\n

However, the LLM might respond better to a prompt in which the list is formatted more clearly or the items are numbered. The BulletedList, NumberedList, BulletedDict and NumberedDict classes are provided to enable this.

For example, to modify the above prompt to contain a numbered list of the items, the NumberedList class can be used. This behaves exactly like a regular Python list except for how it appears when inserted into a formatted string. This class can also be used as the type annotation for items parameter to ensure that this prompt always contains a numbered list.

from magentic import prompt\nfrom magentic.formatting import NumberedList\n\n\n@prompt(\"Continue the list:\\n{items}\")\ndef get_next_items(items: NumberedList[str]) -> list[str]: ...\n\n\nitems = NumberedList([\"apple\", \"banana\", \"cherry\"])\nprint(get_next_items.format(items=items))\n# Continue the list:\n# 1. apple\n# 2. banana\n# 3. cherry\n
"},{"location":"formatting/#custom-formatting","title":"Custom Formatting","text":"

When objects are inserted into formatted strings in Python, the __format__ method is called. By defining or modifying this method you can control how an object is converted to a string in the prompt. If you own the class you can modify the __format__ method directly. Otherwise for third-party classes you will need to create a subcless.

Here's an example of how to represent a dictionary as a bulleted list.

from typing import TypeVar\n\nfrom magentic import prompt\n\nK = TypeVar(\"K\")\nV = TypeVar(\"V\")\n\n\nclass BulletedDict(dict[K, V]):\n    def __format__(self, format_spec: str) -> str:\n        # Here, you could use 'format_spec' to customize the formatting further if needed\n        return \"\\n\".join(f\"- {key}: {value}\" for key, value in self.items())\n\n\n@prompt(\"Identify the odd one out:\\n{items}\")\ndef find_odd_one_out(items: BulletedDict[str, str]) -> str: ...\n\n\nitems = BulletedDict({\"sky\": \"blue\", \"grass\": \"green\", \"sun\": \"purple\"})\nprint(find_odd_one_out.format(items))\n# Identify the odd one out:\n# - sky: blue\n# - grass: green\n# - sun: purple\n
"},{"location":"function-calling/","title":"Function Calling","text":"

For many use cases, it is useful to provide the LLM with tools that it can choose when and how to use. In magentic this is done by passing a list of Python functions to the functions argument of a magentic decorator.

If the LLM chooses to call a function, the decorated function will return a FunctionCall instance. This object can be called to execute the function with the arguments that the LLM provided.

from typing import Literal\n\nfrom magentic import prompt, FunctionCall\n\n\ndef search_twitter(query: str, category: Literal[\"latest\", \"people\"]) -> str:\n    \"\"\"Searches Twitter for a query.\"\"\"\n    print(f\"Searching Twitter for {query!r} in category {category!r}\")\n    return \"<twitter results>\"\n\n\ndef search_youtube(query: str, channel: str = \"all\") -> str:\n    \"\"\"Searches YouTube for a query.\"\"\"\n    print(f\"Searching YouTube for {query!r} in channel {channel!r}\")\n    return \"<youtube results>\"\n\n\n@prompt(\n    \"Use the appropriate search function to answer: {question}\",\n    functions=[search_twitter, search_youtube],\n)\ndef perform_search(question: str) -> FunctionCall[str]: ...\n\n\noutput = perform_search(\"What is the latest news on LLMs?\")\nprint(output)\n# > FunctionCall(<function search_twitter at 0x10c367d00>, 'LLMs', 'latest')\noutput()\n# > Searching Twitter for 'Large Language Models news' in category 'latest'\n# '<twitter results>'\n
"},{"location":"function-calling/#functioncall","title":"FunctionCall","text":"

A FunctionCall combines a function with a set of arguments, ready to be called with no additional inputs required. In magentic, each time the LLM chooses to invoke a function a FunctionCall instance is returned. This allows the chosen function and supplied arguments to be validated or logged before the function is executed.

from magentic import FunctionCall\n\n\ndef plus(a: int, b: int) -> int:\n    return a + b\n\n\nplus_1_2 = FunctionCall(plus, 1, b=2)\nprint(plus_1_2.function)\n# > <function plus at 0x10c39cd30>\nprint(plus_1_2.arguments)\n# > {'a': 1, 'b': 2}\nplus_1_2()\n# 3\n
"},{"location":"function-calling/#prompt_chain","title":"@prompt_chain","text":"

In some cases, you need the model to perform multiple function calls to reach a final answer. The @prompt_chain decorator will execute function calls automatically, append the result to the list of messages, and query the LLM again until a final answer is reached.

In the following example, when describe_weather is called the LLM first calls the get_current_weather function, then uses the result of this to formulate its final answer which gets returned.

from magentic import prompt_chain\n\n\ndef get_current_weather(location, unit=\"fahrenheit\"):\n    \"\"\"Get the current weather in a given location\"\"\"\n    # Pretend to query an API\n    return {\n        \"location\": location,\n        \"temperature\": \"72\",\n        \"unit\": unit,\n        \"forecast\": [\"sunny\", \"windy\"],\n    }\n\n\n@prompt_chain(\n    \"What's the weather like in {city}?\",\n    functions=[get_current_weather],\n)\ndef describe_weather(city: str) -> str: ...\n\n\ndescribe_weather(\"Boston\")\n# 'The current weather in Boston is 72\u00b0F and it is sunny and windy.'\n

LLM-powered functions created using @prompt, @chatprompt and @prompt_chain can be supplied as functions to other @prompt/@prompt_chain decorators, just like regular python functions!

"},{"location":"function-calling/#parallelfunctioncall","title":"ParallelFunctionCall","text":"

The most recent LLMs support \"parallel function calling\". This allows the model to call multiple functions at once. These functions can be executed concurrently, avoiding having to make several serial queries to the model.

You can use ParallelFunctionCall (and AsyncParallelFunctionCall) as a return annotation to indicate that you expect the LLM to make one or more function calls. The returned ParallelFunctionCall is a container of FunctionCall instances. When called, it returns a tuple of their results.

from typing import Literal\n\nfrom magentic import prompt, ParallelFunctionCall\n\n\ndef search_twitter(query: str, category: Literal[\"latest\", \"people\"]) -> str:\n    \"\"\"Searches Twitter for a query.\"\"\"\n    print(f\"Searching Twitter for {query!r} in category {category!r}\")\n    return \"<twitter results>\"\n\n\ndef search_youtube(query: str, channel: str = \"all\") -> str:\n    \"\"\"Searches YouTube for a query.\"\"\"\n    print(f\"Searching YouTube for {query!r} in channel {channel!r}\")\n    return \"<youtube results>\"\n\n\n@prompt(\n    \"Use the appropriate search functions to answer: {question}\",\n    functions=[search_twitter, search_youtube],\n)\ndef perform_search(question: str) -> ParallelFunctionCall[str]: ...\n\n\noutput = perform_search(\"What is the latest news on LLMs?\")\nprint(list(output))\n# > [FunctionCall(<function search_twitter at 0x10c39f760>, 'LLMs', 'latest'),\n#    FunctionCall(<function search_youtube at 0x10c39f7f0>, 'LLMs')]\noutput()\n# > Searching Twitter for 'LLMs' in category 'latest'\n# > Searching YouTube for 'LLMs' in channel 'all'\n# ('<twitter results>', '<youtube results>')\n
"},{"location":"function-calling/#parallelfunctioncall-with-chatprompt","title":"ParallelFunctionCall with @chatprompt","text":"

As with FunctionCall and Pydantic/Python objects, ParallelFunctionCall can be used with @chatprompt for few-shot prompting. In other words, to demonstrate to the LLM how/when it should use functions.

from magentic import (\n    chatprompt,\n    AssistantMessage,\n    FunctionCall,\n    FunctionResultMessage,\n    ParallelFunctionCall,\n    UserMessage,\n)\n\n\ndef plus(a: int, b: int) -> int:\n    return a + b\n\n\ndef minus(a: int, b: int) -> int:\n    return a - b\n\n\nplus_1_2 = FunctionCall(plus, 1, 2)\nminus_2_1 = FunctionCall(minus, 2, 1)\n\n\n@chatprompt(\n    UserMessage(\n        \"Sum 1 and 2. Also subtract 1 from 2.\",\n    ),\n    AssistantMessage(ParallelFunctionCall([plus_1_2, minus_2_1])),\n    FunctionResultMessage(3, plus_1_2),\n    FunctionResultMessage(1, minus_2_1),\n    UserMessage(\"Now add 4 to both results.\"),\n    functions=[plus, minus],\n)\ndef do_math() -> ParallelFunctionCall[int]: ...\n\n\noutput = do_math()\nprint(list(output))\n# > [FunctionCall(<function plus at 0x10c3584c0>, 3, 4),\n#    FunctionCall(<function plus at 0x10c3584c0>, 1, 4)]\noutput()\n# (7, 5)\n
"},{"location":"function-calling/#annotated-parameters","title":"Annotated Parameters","text":"

Like with BaseModel, you can use pydantic's Field to provide additional information for individual function parameters, such as a description. Here's how you could document for the model that the temperature parameter of the activate_oven function is measured in Fahrenheit and should be less than 500.

from typing import Annotated, Literal\n\nfrom pydantic import Field\n\n\ndef activate_oven(\n    temperature: Annotated[int, Field(description=\"Temp in Fahrenheit\", lt=500)],\n    mode: Literal[\"broil\", \"bake\", \"roast\"],\n) -> str:\n    \"\"\"Turn the oven on with the provided settings.\"\"\"\n    return f\"Preheating to {temperature} F with mode {mode}\"\n
"},{"location":"function-calling/#configdict","title":"ConfigDict","text":"

Also like with BaseModel, pydantic's (or magentic's) ConfigDict can be used with functions to configure behavior. Under the hood, the function gets converted to a pydantic model, with every function parameter becoming a field on that model. See the Structured Outputs docs page for more information including the list of configuration options added by magentic.

from typing import Annotated, Literal\n\nfrom magentic import ConfigDict, with_config\nfrom pydantic import Field\n\n\n@with_config(ConfigDict(openai_strict=True))\ndef activate_oven(\n    temperature: Annotated[int, Field(description=\"Temp in Fahrenheit\", lt=500)],\n    mode: Literal[\"broil\", \"bake\", \"roast\"],\n) -> str:\n    \"\"\"Turn the oven on with the provided settings.\"\"\"\n    return f\"Preheating to {temperature} F with mode {mode}\"\n
"},{"location":"logging-and-tracing/","title":"Logging and Tracing","text":"

magentic is instrumented for logging and tracing using Pydantic Logfire. This also makes it compatible with OpenTelemetry.

"},{"location":"logging-and-tracing/#logging-to-stdout","title":"Logging to stdout","text":"

To print magentic spans and logs to stdout, first install the logfire Python package.

pip install logfire\n

Then configure this as needed. See the Pydantic Logfire Integrations docs for the full list of integrations and how to configure these.

import logfire\n\nlogfire.configure(send_to_logfire=False)\nlogfire.instrument_openai()  # optional, to trace OpenAI API calls\n# logfire.instrument_anthropic()  # optional, to trace Anthropic API calls\n

Now when running magentic code, all spans and logs will be printed.

23:02:34.197 Calling prompt-chain describe_weather\n23:02:34.221   Chat Completion with 'gpt-4o' [LLM]\n23:02:35.364   streaming response from 'gpt-4o' took 0.05s [LLM]\n23:02:35.365   Executing function call get_current_weather\n23:02:35.399   Chat Completion with 'gpt-4o' [LLM]\n23:02:35.992   streaming response from 'gpt-4o' took 0.12s [LLM]\n
"},{"location":"logging-and-tracing/#using-pydantic-logfire","title":"Using Pydantic Logfire","text":"

Create a Pydantic Logfire account following their First Steps Guide.

After authenticating locally and creating a project, configure logfire

import logfire\n\nlogfire.configure()  # note: `send_to_logfire` removed. This defaults to True\nlogfire.instrument_openai()  # optional, to trace OpenAI API calls\n# logfire.instrument_anthropic()  # optional, to trace Anthropic API calls\n

Now calls to magentic prompt-functions, prompt-chains etc. will become visible in the Logfire UI.

"},{"location":"logging-and-tracing/#configuring-opentelemetry","title":"Configuring OpenTelemetry","text":"

To enable instrumentation for use with OpenTelemetry, use the following logfire configuration

import logfire\n\nlogfire.configure(\n    send_to_logfire=False,\n    service_name=\"magentic-test\",  # optional, can be set using OTEL_SERVICE_NAME env var\n)\nlogfire.instrument_openai()  # optional, to trace OpenAI API calls\n# logfire.instrument_anthropic()  # optional, to trace Anthropic API calls\n

Now logs and traces for magentic (and OpenAI, Anthropic, ...) will be available to any OTEL tracers.

"},{"location":"logging-and-tracing/#viewing-traces-locally","title":"Viewing Traces Locally","text":"

To view traces locally you can use Jaeger.

First start the Jaeger all-in-one docker container

docker run --rm --name jaeger \\\n  -p 16686:16686 \\\n  -p 4317:4317 \\\n  -p 4318:4318 \\\n  jaegertracing/all-in-one:1.58\n

Then navigate to http://localhost:16686 to access the Jaeger UI. See the Jaeger Getting Started Guide for up-to-date instructions.

Next, install the required OpenTelemetry exporter and configure OpenTelemetry to use this to send traces to Jaeger.

pip install opentelemetry-exporter-otlp\n
from opentelemetry import trace\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\n\ntrace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))\n

Now, traces for magentic code run locally will be visible in the Jaeger UI.

"},{"location":"logging-and-tracing/#enabling-debug-logging","title":"Enabling Debug Logging","text":"

The neatest way to view the raw requests sent to LLM provider APIs is to use Logfire as described above. Another method is to enable debug logs for the LLM provider's Python package. The openai and anthropic packages use the standard library logger and expose an environment variable to set the log level. See the Logging section of the openai README or Logging section of the anthropic README for more information.

Here's an example of what the debug log contains for a simple magentic prompt-function (formatted to improve readability).

import logging\n\nfrom magentic import prompt\n\nlogging.basicConfig(level=logging.DEBUG)\n\n\ndef plus(a: int, b: int) -> int:\n    return a + b\n\n\n@prompt(\n    \"Say hello {n} times\",\n    functions=[plus],\n)\ndef say_hello(n: int) -> str: ...\n\n\nsay_hello(2)\n# ...\n# > DEBUG:openai._base_client:Request options: {\n#     \"method\": \"post\",\n#     \"url\": \"/chat/completions\",\n#     \"files\": None,\n#     \"json_data\": {\n#         \"messages\": [{\"role\": \"user\", \"content\": \"Say hello 2 times\"}],\n#         \"model\": \"gpt-3.5-turbo\",\n#         \"functions\": [\n#             {\n#                 \"name\": \"plus\",\n#                 \"parameters\": {\n#                     \"properties\": {\n#                         \"a\": {\"title\": \"A\", \"type\": \"integer\"},\n#                         \"b\": {\"title\": \"B\", \"type\": \"integer\"},\n#                     },\n#                     \"required\": [\"a\", \"b\"],\n#                     \"type\": \"object\",\n#                 },\n#             }\n#         ],\n#         \"max_tokens\": None,\n#         \"stream\": True,\n#         \"temperature\": None,\n#     },\n# }\n# ...\n
"},{"location":"retrying/","title":"Retrying","text":""},{"location":"retrying/#llm-assisted-retries","title":"LLM-Assisted Retries","text":"

Occasionally the LLM returns an output that cannot be parsed into any of the output types or function calls that were requested. Additionally, the pydantic models you define might have extra validation that is not represented by the type annotations alone. In these cases, LLM-assisted retries can be used to automatically resubmit the output as well as the associated error message back to the LLM, giving it another opportunity with more information to meet the output schema requirements.

To enable retries, simply set the max_retries parameter to a non-zero value in @prompt or @chatprompt.

In this example

  • the LLM first returns a country that is not Ireland
  • then the pydantic model validation fails with error \"Country must be Ireland\"
  • the original output as well as a message containing the error are resubmitted to the LLM
  • the LLM correctly meets the output requirement returning \"Ireland\"
from typing import Annotated\n\nfrom magentic import prompt\nfrom pydantic import AfterValidator, BaseModel\n\n\ndef assert_is_ireland(v: str) -> str:\n    if v != \"Ireland\":\n        raise ValueError(\"Country must be Ireland\")\n    return v\n\n\nclass Country(BaseModel):\n    name: Annotated[str, AfterValidator(assert_is_ireland)]\n    capital: str\n\n\n@prompt(\n    \"Return a country\",\n    max_retries=3,\n)\ndef get_country() -> Country: ...\n\n\nget_country()\n# 05:13:55.607 Calling prompt-function get_country\n# 05:13:55.622   LLM-assisted retries enabled. Max 3\n# 05:13:55.627     Chat Completion with 'gpt-4o' [LLM]\n# 05:13:56.309     streaming response from 'gpt-4o' took 0.11s [LLM]\n# 05:13:56.310     Retrying Chat Completion. Attempt 1.\n# 05:13:56.322     Chat Completion with 'gpt-4o' [LLM]\n# 05:13:57.456     streaming response from 'gpt-4o' took 0.00s [LLM]\n#\n# Country(name='Ireland', capital='Dublin')\n

LLM-Assisted retries are intended to address cases where the LLM failed to generate valid output. Errors due to LLM provider rate limiting, internet connectivity issues, or other issues that cannot be solved by reprompting the LLM should be handled using other methods. For example jd/tenacity or hynek/stamina to retry a Python function.

"},{"location":"retrying/#retrychatmodel","title":"RetryChatModel","text":"

Under the hood, LLM-assisted retries are implemented using the RetryChatModel which wraps any other ChatModel, catches exceptions, and resubmits them to the LLM. To implement your own retry handling you can follow the pattern of this class. Please file a GitHub issue if you encounter exceptions that should be included in the LLM-assisted retries.

To use the RetryChatModel directly rather than via the max_retries parameter, simply pass it as the model argument to the decorator. Extending the example above

from magentic import OpenaiChatModel\nfrom magentic.chat_model.retry_chat_model import RetryChatModel\n\n\n@prompt(\n    \"Return a country\",\n    model=RetryChatModel(OpenaiChatModel(\"gpt-4o-mini\"), max_retries=3),\n)\ndef get_country() -> Country: ...\n\n\nget_country()\n
"},{"location":"streaming/","title":"Streaming","text":"

The StreamedStr (and AsyncStreamedStr) class can be used to stream the output of the LLM. This allows you to process the text while it is being generated, rather than receiving the whole output at once.

from magentic import prompt, StreamedStr\n\n\n@prompt(\"Tell me about {country}\")\ndef describe_country(country: str) -> StreamedStr: ...\n\n\n# Print the chunks while they are being received\nfor chunk in describe_country(\"Brazil\"):\n    print(chunk, end=\"\")\n# 'Brazil, officially known as the Federative Republic of Brazil, is ...'\n

Multiple StreamedStr can be created at the same time to stream LLM outputs concurrently. In the below example, generating the description for multiple countries takes approximately the same amount of time as for a single country.

from time import time\n\ncountries = [\"Australia\", \"Brazil\", \"Chile\"]\n\n\n# Generate the descriptions one at a time\nstart_time = time()\nfor country in countries:\n    # Converting `StreamedStr` to `str` blocks until the LLM output is fully generated\n    description = str(describe_country(country))\n    print(f\"{time() - start_time:.2f}s : {country} - {len(description)} chars\")\n\n# 22.72s : Australia - 2130 chars\n# 41.63s : Brazil - 1884 chars\n# 74.31s : Chile - 2968 chars\n\n\n# Generate the descriptions concurrently by creating the StreamedStrs at the same time\nstart_time = time()\nstreamed_strs = [describe_country(country) for country in countries]\nfor country, streamed_str in zip(countries, streamed_strs):\n    description = str(streamed_str)\n    print(f\"{time() - start_time:.2f}s : {country} - {len(description)} chars\")\n\n# 22.79s : Australia - 2147 chars\n# 23.64s : Brazil - 2202 chars\n# 24.67s : Chile - 2186 chars\n
"},{"location":"streaming/#object-streaming","title":"Object Streaming","text":"

Structured outputs can also be streamed from the LLM by using the return type annotation Iterable (or AsyncIterable). This allows each item to be processed while the next one is being generated.

from collections.abc import Iterable\nfrom time import time\n\nfrom magentic import prompt\nfrom pydantic import BaseModel\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\n@prompt(\"Create a Superhero team named {name}.\")\ndef create_superhero_team(name: str) -> Iterable[Superhero]: ...\n\n\nstart_time = time()\nfor hero in create_superhero_team(\"The Food Dudes\"):\n    print(f\"{time() - start_time:.2f}s : {hero}\")\n\n# 2.23s : name='Pizza Man' age=30 power='Can shoot pizza slices from his hands' enemies=['The Hungry Horde', 'The Junk Food Gang']\n# 4.03s : name='Captain Carrot' age=35 power='Super strength and agility from eating carrots' enemies=['The Sugar Squad', 'The Greasy Gang']\n# 6.05s : name='Ice Cream Girl' age=25 power='Can create ice cream out of thin air' enemies=['The Hot Sauce Squad', 'The Healthy Eaters']\n
"},{"location":"streaming/#streamedresponse","title":"StreamedResponse","text":"

Some LLMs have the ability to generate text output and make tool calls in the same response. This allows them to perform chain-of-thought reasoning or provide additional context to the user. In magentic, the StreamedResponse (or AsyncStreamedResponse) class can be used to request this type of output. This object is an iterable of StreamedStr (or AsyncStreamedStr) and FunctionCall instances.

Consuming StreamedStr

The StreamedStr object must be iterated over before the next item in the StreamedResponse is processed, otherwise the string output will be lost. This is because the StreamedResponse and StreamedStr share the same underlying generator, so advancing the StreamedResponse iterator skips over the StreamedStr items. The StreamedStr object has internal caching so after iterating over it once the chunks will remain available.

In the example below, we request that the LLM generates a greeting and then calls a function to get the weather for two cities. The StreamedResponse object is then iterated over to print the output, and the StreamedStr and FunctionCall items are processed separately.

from magentic import prompt, FunctionCall, StreamedResponse, StreamedStr\n\n\ndef get_weather(city: str) -> str:\n    return f\"The weather in {city} is 20\u00b0C.\"\n\n\n@prompt(\n    \"Say hello, then get the weather for: {cities}\",\n    functions=[get_weather],\n)\ndef describe_weather(cities: list[str]) -> StreamedResponse: ...\n\n\nresponse = describe_weather([\"Cape Town\", \"San Francisco\"])\nfor item in response:\n    if isinstance(item, StreamedStr):\n        for chunk in item:\n            # print the chunks as they are received\n            print(chunk, sep=\"\", end=\"\")\n        print()\n    if isinstance(item, FunctionCall):\n        # print the function call, then call it and print the result\n        print(item)\n        print(item())\n\n# Hello! I'll get the weather for Cape Town and San Francisco for you.\n# FunctionCall(<function get_weather at 0x1109825c0>, 'Cape Town')\n# The weather in Cape Town is 20\u00b0C.\n# FunctionCall(<function get_weather at 0x1109825c0>, 'San Francisco')\n# The weather in San Francisco is 20\u00b0C.\n
"},{"location":"structured-outputs/","title":"Structured Outputs","text":""},{"location":"structured-outputs/#pydantic-models","title":"Pydantic Models","text":"

The @prompt decorator will respect the return type annotation of the decorated function. This can be any type supported by pydantic including a pydantic model. See the Pydantic docs for more information about models.

from magentic import prompt\nfrom pydantic import BaseModel\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\n@prompt(\"Create a Superhero named {name}.\")\ndef create_superhero(name: str) -> Superhero: ...\n\n\ncreate_superhero(\"Garden Man\")\n# Superhero(name='Garden Man', age=30, power='Control over plants', enemies=['Pollution Man', 'Concrete Woman'])\n
"},{"location":"structured-outputs/#using-field","title":"Using Field","text":"

With pydantic's BaseModel, you can use Field to provide additional information for individual attributes, such as a description.

from magentic import prompt\nfrom pydantic import BaseModel\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int = Field(\n        description=\"The age of the hero, which could be much older than humans.\"\n    )\n    power: str = Field(examples=[\"Runs really fast\"])\n    enemies: list[str]\n\n\n@prompt(\"Create a Superhero named {name}.\")\ndef create_superhero(name: str) -> Superhero: ...\n
"},{"location":"structured-outputs/#configdict","title":"ConfigDict","text":"

Pydantic also supports configuring the BaseModel by setting the model_config attribute. Magentic extends pydantic's ConfigDict class to add the following additional configuration options

  • openai_strict: bool Indicates whether to use OpenAI's Structured Outputs feature.

See the pydantic Configuration docs for the inherited configuration options.

from magentic import prompt, ConfigDict\nfrom pydantic import BaseModel\n\n\nclass Superhero(BaseModel):\n    model_config = ConfigDict(openai_strict=True)\n\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\n@prompt(\"Create a Superhero named {name}.\")\ndef create_superhero(name: str) -> Superhero: ...\n\n\ncreate_superhero(\"Garden Man\")\n
"},{"location":"structured-outputs/#json-schema","title":"JSON Schema","text":"

OpenAI Structured Outputs

Setting openai_strict=True results in a different JSON schema than that from .model_json_schema() being sent to the LLM. Use openai.pydantic_function_tool(Superhero) to generate the JSON schema in this case.

You can generate the JSON schema for the pydantic model using the .model_json_schema() method. This is what is sent to the LLM.

Running Superhero.model_json_schema() for the above definition reveals the following JSON schema

{\n    \"properties\": {\n        \"name\": {\"title\": \"Name\", \"type\": \"string\"},\n        \"age\": {\n            \"description\": \"The age of the hero, which could be much older than humans.\",\n            \"title\": \"Age\",\n            \"type\": \"integer\",\n        },\n        \"power\": {\n            \"examples\": [\"Runs really fast\"],\n            \"title\": \"Power\",\n            \"type\": \"string\",\n        },\n        \"enemies\": {\"items\": {\"type\": \"string\"}, \"title\": \"Enemies\", \"type\": \"array\"},\n    },\n    \"required\": [\"name\", \"age\", \"power\", \"enemies\"],\n    \"title\": \"Superhero\",\n    \"type\": \"object\",\n}\n

If a StructuredOutputError is raised often, this indicates that the LLM is failing to match the schema. The traceback for these errors includes the underlying pydantic ValidationError which shows in what way the received response was invalid. To combat these errors there are several options

  • Add descriptions or examples for individual fields to demonstrate valid values.
  • Simplify the output schema, including using more flexible types (e.g. str instead of datetime) or allowing fields to be nullable with | None.
  • Switch to a \"more intelligent\" LLM. See Configuration for how to do this.
"},{"location":"structured-outputs/#python-types","title":"Python Types","text":"

Regular Python types can also be used as the function return type.

from magentic import prompt\nfrom pydantic import BaseModel, Field\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\ngarden_man = Superhero(\n    name=\"Garden Man\",\n    age=30,\n    power=\"Control over plants\",\n    enemies=[\"Pollution Man\", \"Concrete Woman\"],\n)\n\n\n@prompt(\"Return True if {hero.name} will be defeated by enemies {hero.enemies}\")\ndef will_be_defeated(hero: Superhero) -> bool: ...\n\n\nhero_defeated = will_be_defeated(garden_man)\nprint(hero_defeated)\n# > True\n
"},{"location":"structured-outputs/#chain-of-thought-prompting","title":"Chain-of-Thought Prompting","text":"

StreamedResponse

It is now recommended to use StreamedResponse for chain-of-thought prompting, as this uses the LLM provider's native chain-of-thought capabilities. See StreamedResponse for more information.

Using a simple Python type as the return annotation might result in poor results as the LLM has no time to arrange its thoughts before answering. To allow the LLM to work through this \"chain of thought\" you can instead return a pydantic model with initial fields for explaining the final response.

from magentic import prompt\nfrom pydantic import BaseModel, Field\n\n\nclass ExplainedDefeated(BaseModel):\n    explanation: str = Field(\n        description=\"Describe the battle between the hero and their enemy.\"\n    )\n    defeated: bool = Field(description=\"True if the hero was defeated.\")\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\n@prompt(\"Return True if {hero.name} will be defeated by enemies {hero.enemies}\")\ndef will_be_defeated(hero: Superhero) -> ExplainedDefeated: ...\n\n\ngarden_man = Superhero(\n    name=\"Garden Man\",\n    age=30,\n    power=\"Control over plants\",\n    enemies=[\"Pollution Man\", \"Concrete Woman\"],\n)\n\nhero_defeated = will_be_defeated(garden_man)\nprint(hero_defeated.defeated)\n# > True\nprint(hero_defeated.explanation)\n# > 'Garden Man is an environmental hero who fights against Pollution Man ...'\n
"},{"location":"structured-outputs/#explained","title":"Explained","text":"

Using chain-of-thought is a common approach to improve the output of the model, so a generic Explained model might be generally useful. The description or example parameters of Field can be used to demonstrate the desired style and detail of the explanations.

from typing import Generic, TypeVar\n\nfrom magentic import prompt\nfrom pydantic import BaseModel, Field\n\n\nT = TypeVar(\"T\")\n\n\nclass Explained(BaseModel, Generic[T]):\n    explanation: str = Field(description=\"Explanation of how the value was determined.\")\n    value: T\n\n\n@prompt(\"Return True if {hero.name} will be defeated by enemies {hero.enemies}\")\ndef will_be_defeated(hero: Superhero) -> Explained[bool]: ...\n
"},{"location":"type-checking/","title":"Type Checking","text":"

Many type checkers will raise warnings or errors for functions with the @prompt decorator due to the function having no body or return value. There are several ways to deal with these.

  1. Disable the check globally for the type checker. For example in mypy by disabling error code empty-body. 
    # pyproject.toml\n[tool.mypy]\ndisable_error_code = [\"empty-body\"]\n
  2. Make the function body ... (this does not satisfy mypy) or raise. 
    @prompt(\"Choose a color\")\ndef random_color() -> str: ...\n
  3. Use comment # type: ignore[empty-body] on each function. In this case you can add a docstring instead of .... 
    @prompt(\"Choose a color\")\ndef random_color() -> str:  # type: ignore[empty-body]\n    \"\"\"Returns a random color.\"\"\"\n
"},{"location":"vision/","title":"Vision","text":"

Image inputs can be provided to LLMs in magentic by using the UserImageMessage message type.

Anthropic Image URLs

Anthropic models currently do not support supplying an image as a url, just bytes.

For more information visit the OpenAI Vision API documentation or the Anthropic Vision API documentation.

"},{"location":"vision/#userimagemessage","title":"UserImageMessage","text":"

The UserImageMessage can be used in @chatprompt alongside other messages. The LLM must be set to an OpenAI or Anthropic model that supports vision, for example gpt-4o (the default ChatModel). This can be done by passing the model parameter to @chatprompt, or through the other methods of configuration.

from pydantic import BaseModel, Field\n\nfrom magentic import chatprompt, UserMessage\nfrom magentic.vision import UserImageMessage\n\n\nIMAGE_URL_WOODEN_BOARDWALK = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\"\n\n\nclass ImageDetails(BaseModel):\n    description: str = Field(description=\"A brief description of the image.\")\n    name: str = Field(description=\"A short name.\")\n\n\n@chatprompt(\n    UserMessage(\"Describe the following image in one sentence.\"),\n    UserImageMessage(IMAGE_URL_WOODEN_BOARDWALK),\n)\ndef describe_image() -> ImageDetails: ...\n\n\nimage_details = describe_image()\nprint(image_details.name)\n# 'Wooden Boardwalk in Green Wetland'\nprint(image_details.description)\n# 'A serene wooden boardwalk meanders through a lush green wetland under a blue sky dotted with clouds.'\n

For more info on the @chatprompt decorator, see Chat Prompting.

"},{"location":"vision/#placeholder","title":"Placeholder","text":"

In the previous example, the image url was tied to the function. To provide the image as a function parameter, use Placeholder. This substitutes a function argument into the message when the function is called.

from magentic import chatprompt, Placeholder, UserMessage\nfrom magentic.vision import UserImageMessage\n\n\nIMAGE_URL_WOODEN_BOARDWALK = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\"\n\n\n@chatprompt(\n    UserMessage(\"Describe the following image in one sentence.\"),\n    UserImageMessage(Placeholder(str, \"image_url\")),\n)\ndef describe_image(image_url: str) -> str: ...\n\n\ndescribe_image(IMAGE_URL_WOODEN_BOARDWALK)\n# 'A wooden boardwalk meanders through lush green wetlands under a partly cloudy blue sky.'\n
"},{"location":"vision/#bytes","title":"bytes","text":"

UserImageMessage can also accept bytes as input. Like str, this can be passed directly or via Placeholder.

import requests\n\nfrom magentic import chatprompt, Placeholder, UserMessage\nfrom magentic.vision import UserImageMessage\n\n\nIMAGE_URL_WOODEN_BOARDWALK = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\"\n\n\ndef url_to_bytes(url: str) -> bytes:\n    \"\"\"Get the content of a URL as bytes.\"\"\"\n\n    # A custom user-agent is necessary to comply with Wikimedia user-agent policy\n    # https://meta.wikimedia.org/wiki/User-Agent_policy\n    headers = {\"User-Agent\": \"MagenticExampleBot (https://magentic.dev/)\"}\n    return requests.get(url, headers=headers, timeout=10).content\n\n\n@chatprompt(\n    UserMessage(\"Describe the following image in one sentence.\"),\n    UserImageMessage(Placeholder(bytes, \"image_bytes\")),\n)\ndef describe_image(image_bytes: bytes) -> str: ...\n\n\nimage_bytes = url_to_bytes(IMAGE_URL_WOODEN_BOARDWALK)\ndescribe_image(image_bytes)\n# 'The image shows a wooden boardwalk extending through a lush green wetland with a backdrop of blue skies and scattered clouds.'\n
"},{"location":"examples/chain_of_verification/","title":"Chain of Verification (CoVe)","text":"In\u00a0[1]: Copied! 
# Define the prompts\n\nimport asyncio\n\nfrom magentic import prompt\n\n\n@prompt(\"{query}\")\nasync def answer_query(query: str) -> str: ...\n\n\n@prompt(\n    \"\"\"\\\nQuery: {query}\nResponse: {response}\n\nProvide specific questions to verify the facts in the above response as related to the query.\n\"\"\"\n)\nasync def generate_verification_questions(query: str, response: str) -> list[str]: ...\n\n\n@prompt(\n    \"\"\"\\\n{context}\n\nGiven the above context, what is the answer to the following question?\n\n{query}\"\"\"\n)\nasync def answer_query_with_context(query: str, context: str) -> str: ...\n
# Define the prompts import asyncio from magentic import prompt @prompt(\"{query}\") async def answer_query(query: str) -> str: ... @prompt( \"\"\"\\ Query: {query} Response: {response} Provide specific questions to verify the facts in the above response as related to the query. \"\"\" ) async def generate_verification_questions(query: str, response: str) -> list[str]: ... @prompt( \"\"\"\\ {context} Given the above context, what is the answer to the following question? {query}\"\"\" ) async def answer_query_with_context(query: str, context: str) -> str: ... In\u00a0[2]: Copied! 
# 1. Generate Baseline Response\n# Given a query, generate the response using the LLM.\n\nquery = \"Name some politicians who were born in NY, New York\"\n\nbaseline_response = await answer_query(query)\nprint(baseline_response)\n
# 1. Generate Baseline Response # Given a query, generate the response using the LLM. query = \"Name some politicians who were born in NY, New York\" baseline_response = await answer_query(query) print(baseline_response) 
Here are some politicians who were born in New York, New York:\n\n1. Franklin D. Roosevelt - 32nd President of the United States.\n2. Theodore Roosevelt - 26th President of the United States.\n3. Donald Trump - 45th President of the United States.\n4. Hillary Clinton - Former Secretary of State and Democratic nominee for President in 2016.\n5. Michael Bloomberg - Former Mayor of New York City and businessman.\n6. Rudy Giuliani - Former Mayor of New York City and attorney.\n7. Chuck Schumer - U.S. Senator from New York and current Senate Majority Leader.\n8. Kirsten Gillibrand - U.S. Senator from New York.\n9. Mario Cuomo - Former Governor of New York.\n10. Andrew Cuomo - Current Governor of New York.\n\nPlease note that this is not an exhaustive list, and there are many more politicians who were born in New York, New York.\n
In\u00a0[3]: Copied! 
# 2. Plan Verifications\n# Given both query and baseline response, generate a list of verification questions\n# that could help to self-analyze if there are any mistakes in the original response.\n\nverification_questions = await generate_verification_questions(query, baseline_response)\n\nfor q in verification_questions:\n    print(q)\n
# 2. Plan Verifications # Given both query and baseline response, generate a list of verification questions # that could help to self-analyze if there are any mistakes in the original response. verification_questions = await generate_verification_questions(query, baseline_response) for q in verification_questions: print(q) 
Was Franklin D. Roosevelt born in New York, New York?\nWas Theodore Roosevelt born in New York, New York?\nWas Donald Trump born in New York, New York?\nWas Hillary Clinton born in New York, New York?\nWas Michael Bloomberg born in New York, New York?\nWas Rudy Giuliani born in New York, New York?\nWas Chuck Schumer born in New York, New York?\nWas Kirsten Gillibrand born in New York, New York?\nWas Mario Cuomo born in New York, New York?\nIs Andrew Cuomo the current Governor of New York?\n
In\u00a0[4]: Copied! 
# 3. Execute Verifications\n# Answer each verification question in turn, and hence check the answer against the\n# original response to check for inconsistencies or mistakes.\n\nverification_answers = await asyncio.gather(\n    *(answer_query(question) for question in verification_questions)\n)\n\nfor ans in verification_answers:\n    print(ans)\n
# 3. Execute Verifications # Answer each verification question in turn, and hence check the answer against the # original response to check for inconsistencies or mistakes. verification_answers = await asyncio.gather( *(answer_query(question) for question in verification_questions) ) for ans in verification_answers: print(ans) 
Yes, Franklin D. Roosevelt was born on January 30, 1882, in Hyde Park, New York, which is located in Dutchess County.\nYes, Theodore Roosevelt was born in New York City, New York on October 27, 1858. Specifically, he was born in a house located at 28 East 20th Street in Manhattan.\nYes, Donald Trump was indeed born in New York, New York on June 14, 1946.\nNo, Hillary Clinton was not born in New York, New York. She was born on October 26, 1947, in Chicago, Illinois.\nNo, Michael Bloomberg was born in Boston, Massachusetts on February 14, 1942.\nYes, Rudy Giuliani was born in New York, New York. He was born on May 28, 1944, in the Brooklyn borough of New York City.\nYes, Chuck Schumer was born in Brooklyn, New York on November 23, 1950.\nNo, Kirsten Gillibrand was born in Albany, New York on December 9, 1966.\nYes, Mario Cuomo was born in New York City, New York, United States. He was born on June 15, 1932, in the borough of Queens, specifically in the neighborhood of South Jamaica.\nAs of September 2021, Andrew Cuomo is the current Governor of New York. However, please note that political positions can change, and it is always recommended to verify the information with up-to-date sources.\n
In\u00a0[5]: Copied! 
# 4. Generate Final Verified Response\n# Given the discovered inconsistencies (if any), generate a revised response\n# incorporating the verification results.\n\ncontext = \"\\n\".join(verification_answers)\nverified_response = await answer_query_with_context(query, context)\nprint(verified_response)\n
# 4. Generate Final Verified Response # Given the discovered inconsistencies (if any), generate a revised response # incorporating the verification results. context = \"\\n\".join(verification_answers) verified_response = await answer_query_with_context(query, context) print(verified_response) 
Some politicians who were born in New York, New York include Franklin D. Roosevelt, Theodore Roosevelt, Donald Trump, Rudy Giuliani, and Mario Cuomo.\n
"},{"location":"examples/chain_of_verification/#chain-of-verification-cove","title":"Chain of Verification (CoVe)\u00b6","text":"

This notebook is a basic implementation of the paper Chain-of-Verification Reduces Hallucination In Large Language Models (2023) (arXiv: [2309.11495]).

"},{"location":"examples/rag_github/","title":"Retrieval-Augmented Generation with GitHub","text":"In\u00a0[\u00a0]: Copied! 
# Install dependencies (skip this cell if already installed)\n! pip install magentic\n! pip install ghapi\n
# Install dependencies (skip this cell if already installed) ! pip install magentic ! pip install ghapi In\u00a0[2]: Copied! 
# Configure magentic to use the `gpt-3.5-turbo` model for this notebook\n%env MAGENTIC_OPENAI_MODEL=gpt-3.5-turbo\n
# Configure magentic to use the `gpt-3.5-turbo` model for this notebook %env MAGENTIC_OPENAI_MODEL=gpt-3.5-turbo 
env: MAGENTIC_OPENAI_MODEL=gpt-3.5-turbo\n

Let's start by creating a prompt-function to generate some text recommending GitHub repos for a topic.

In\u00a0[3]: Copied! 
# Create a prompt-function to describe the latest GitHub repos\n\nfrom IPython.display import Markdown, display\n\nfrom magentic import prompt\n\n\n@prompt(\n    \"\"\"What are the latest github repos I should use related to {topic}?\n    Recommend three in particular that I should check out and why.\n    Provide a link to each, and a note on whether they are actively maintained.\n    \"\"\"\n)\ndef recommmend_github_repos(topic: str) -> str: ...\n\n\noutput = recommmend_github_repos(\"LLMs\")\ndisplay(Markdown(output))\n
# Create a prompt-function to describe the latest GitHub repos from IPython.display import Markdown, display from magentic import prompt @prompt( \"\"\"What are the latest github repos I should use related to {topic}? Recommend three in particular that I should check out and why. Provide a link to each, and a note on whether they are actively maintained. \"\"\" ) def recommmend_github_repos(topic: str) -> str: ... output = recommmend_github_repos(\"LLMs\") display(Markdown(output))
  1. Hugging Face Transformers: This repository contains a library for Natural Language Processing (NLP) tasks using the latest Transformer models, including LLMs. It is actively maintained by Hugging Face, a popular NLP research group, and has a large community contributing to it.

Link: https://github.com/huggingface/transformers

  1. OpenAI GPT-3: This repository contains the code for OpenAI's GPT-3 model, one of the most advanced LLMs available. While the repository may not be frequently updated due to proprietary restrictions, it provides valuable insights into how state-of-the-art LLMs are implemented.

Link: https://github.com/openai/gpt-3

  1. AllenNLP: AllenNLP is a deep learning library for NLP research that provides easy-to-use tools for building and experimenting with LLMs. The repository is actively maintained by the Allen Institute for AI and offers a wide range of pre-trained models, including BERT and GPT-2.

Link: https://github.com/allenai/allennlp

Please note that the availability and maintenance status of these repositories may change over time, so it's a good idea to check for the latest updates before diving in.

The LLM has no knowledge of GitHub repos created after its knowledge cutoff date! Also, it occasionally hallucinates some of its answers. To solve these issues we need to provide it with up-to-date information in the prompt, which it can use to generate an informed answer.

First we'll create a function for searching for GitHub repos.

In\u00a0[4]: Copied! 
# Create a function to search for GitHub repos\n\nfrom ghapi.all import GhApi\nfrom pydantic import BaseModel\n\ngithub = GhApi(authenticate=False)\n\n\nclass GithubRepo(BaseModel):\n    full_name: str\n    description: str\n    html_url: str\n    stargazers_count: int\n    pushed_at: str\n\n\ndef search_github_repos(query: str, num_results: int = 10):\n    results = github.search.repos(query, per_page=num_results)\n    return [GithubRepo.model_validate(item) for item in results[\"items\"]]\n
# Create a function to search for GitHub repos from ghapi.all import GhApi from pydantic import BaseModel github = GhApi(authenticate=False) class GithubRepo(BaseModel): full_name: str description: str html_url: str stargazers_count: int pushed_at: str def search_github_repos(query: str, num_results: int = 10): results = github.search.repos(query, per_page=num_results) return [GithubRepo.model_validate(item) for item in results[\"items\"]] In\u00a0[5]: Copied! 
# Test that github search works\n\nfor item in search_github_repos(\"openai\", num_results=3):\n    print(item.model_dump_json(indent=2))\n
# Test that github search works for item in search_github_repos(\"openai\", num_results=3): print(item.model_dump_json(indent=2)) 
{\n  \"full_name\": \"openai/openai-cookbook\",\n  \"description\": \"Examples and guides for using the OpenAI API\",\n  \"html_url\": \"https://github.com/openai/openai-cookbook\",\n  \"stargazers_count\": 55805,\n  \"pushed_at\": \"2024-04-19T19:05:02Z\"\n}\n{\n  \"full_name\": \"betalgo/openai\",\n  \"description\": \"OpenAI .NET sdk - Azure OpenAI, ChatGPT, Whisper,  and DALL-E \",\n  \"html_url\": \"https://github.com/betalgo/openai\",\n  \"stargazers_count\": 2721,\n  \"pushed_at\": \"2024-04-20T22:50:28Z\"\n}\n{\n  \"full_name\": \"openai/openai-python\",\n  \"description\": \"The official Python library for the OpenAI API\",\n  \"html_url\": \"https://github.com/openai/openai-python\",\n  \"stargazers_count\": 19786,\n  \"pushed_at\": \"2024-04-21T01:04:42Z\"\n}\n

Now, we can provide the results of the search as context to the LLM to create an improved recommmend_github_repos function.

In\u00a0[6]: Copied! 
# Combine the search with a prompt-function to describe the latest GitHub repos\n\nfrom magentic import prompt\n\n\n@prompt(\n    \"\"\"What are the latest github repos I should use related to {topic}?\n    Recommend three in particular that I should check out and why.\n    Provide a link to each, and a note on whether they are actively maintained.\n\n    Here are the latest search results for this topic on GitHub:\n    {search_results}\n    \"\"\",\n)\ndef recommmend_github_repos_using_search_results(\n    topic: str, search_results: list[GithubRepo]\n) -> str: ...\n\n\ndef recommmend_github_repos(topic: str) -> str:\n    search_results = search_github_repos(topic, num_results=10)\n    return recommmend_github_repos_using_search_results(topic, search_results)\n\n\noutput = recommmend_github_repos(\"LLMs\")\ndisplay(Markdown(output))\n
# Combine the search with a prompt-function to describe the latest GitHub repos from magentic import prompt @prompt( \"\"\"What are the latest github repos I should use related to {topic}? Recommend three in particular that I should check out and why. Provide a link to each, and a note on whether they are actively maintained. Here are the latest search results for this topic on GitHub: {search_results} \"\"\", ) def recommmend_github_repos_using_search_results( topic: str, search_results: list[GithubRepo] ) -> str: ... def recommmend_github_repos(topic: str) -> str: search_results = search_github_repos(topic, num_results=10) return recommmend_github_repos_using_search_results(topic, search_results) output = recommmend_github_repos(\"LLMs\") display(Markdown(output))

Based on the latest search results, here are three GitHub repos related to Large Language Models (LLMs) that you should check out:

  1. gpt4all:

    • Description: gpt4all: run open-source LLMs anywhere
    • Stargazers Count: 63,790
    • Last Pushed: 2024-04-19
    • Active Maintenance: Yes

  2. LLaMA-Factory:

    • Description: Unify Efficient Fine-Tuning of 100+ LLMs
    • Stargazers Count: 17,047
    • Last Pushed: 2024-04-21
    • Active Maintenance: Yes

  3. LLMsPracticalGuide:

    • Description: A curated list of practical guide resources of LLMs (LLMs Tree, Examples, Papers)
    • Stargazers Count: 8,484
    • Last Pushed: 2024-01-10
    • Active Maintenance: It seems less actively maintained compared to the other two repos, but still worth checking out.

These repos cover a range of topics related to LLMs and can provide valuable resources and tools for your projects.

Now the answer contains up-to-date and correct information!

"},{"location":"examples/rag_github/#retrieval-augmented-generation-with-github","title":"Retrieval-Augmented Generation with GitHub\u00b6","text":"

This notebook demonstrates how to perform Retrieval-Augmented Generation (RAG) with magentic using the GitHub API. Essentially, RAG provides context to the LLM which it can use when generating its response. This approach allows us to insert new or private information that was not present in the model's training data.

"},{"location":"examples/registering_custom_type/","title":"Registering a Custom Type","text":"In\u00a0[1]: Copied! 
# Create FunctionSchema for pd.DataFrame\n\nimport json\nfrom collections.abc import Iterable\nfrom typing import Any\n\nimport pandas as pd\n\nfrom magentic.chat_model.function_schema import FunctionSchema, register_function_schema\n\n\n@register_function_schema(pd.DataFrame)\nclass DataFrameFunctionSchema(FunctionSchema[pd.DataFrame]):\n    @property\n    def name(self) -> str:\n        \"\"\"The name of the function.\n\n        Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.\n        \"\"\"\n        return \"dataframe\"\n\n    @property\n    def description(self) -> str | None:\n        return \"A DataFrame object.\"\n\n    @property\n    def parameters(self) -> dict[str, Any]:\n        \"\"\"The parameters the functions accepts as a JSON Schema object.\"\"\"\n        return {\n            \"properties\": {\n                \"columns\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}},\n                \"data\": {\n                    \"type\": \"array\",\n                    \"items\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}},\n                },\n            },\n            \"required\": [\"index\", \"columns\", \"data\"],\n            \"type\": \"object\",\n        }\n\n    def parse_args(self, chunks: Iterable[str]) -> pd.DataFrame:\n        \"\"\"Parse an iterable of string chunks into the function arguments.\"\"\"\n        args = json.loads(\"\".join(chunks))\n        return pd.DataFrame(**args)\n\n    def serialize_args(self, value: pd.DataFrame) -> dict:\n        \"\"\"Serialize an object into a JSON string of function arguments.\"\"\"\n        return {\n            \"columns\": value.columns.tolist(),\n            \"data\": value.to_numpy().tolist(),\n        }\n
# Create FunctionSchema for pd.DataFrame import json from collections.abc import Iterable from typing import Any import pandas as pd from magentic.chat_model.function_schema import FunctionSchema, register_function_schema @register_function_schema(pd.DataFrame) class DataFrameFunctionSchema(FunctionSchema[pd.DataFrame]): @property def name(self) -> str: \"\"\"The name of the function. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. \"\"\" return \"dataframe\" @property def description(self) -> str | None: return \"A DataFrame object.\" @property def parameters(self) -> dict[str, Any]: \"\"\"The parameters the functions accepts as a JSON Schema object.\"\"\" return { \"properties\": { \"columns\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}}, \"data\": { \"type\": \"array\", \"items\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}}, }, }, \"required\": [\"index\", \"columns\", \"data\"], \"type\": \"object\", } def parse_args(self, chunks: Iterable[str]) -> pd.DataFrame: \"\"\"Parse an iterable of string chunks into the function arguments.\"\"\" args = json.loads(\"\".join(chunks)) return pd.DataFrame(**args) def serialize_args(self, value: pd.DataFrame) -> dict: \"\"\"Serialize an object into a JSON string of function arguments.\"\"\" return { \"columns\": value.columns.tolist(), \"data\": value.to_numpy().tolist(), } In\u00a0[2]: Copied! 
# Roundtrip test the new FunctionSchema\n\nfunction_schema = DataFrameFunctionSchema(pd.DataFrame)\n\ndf_test = pd.DataFrame(\n    {\n        \"A\": [1, 2, 3],\n        \"B\": [4, 5, 6],\n    },\n)\n\nargs = function_schema.serialize_args(df_test)\nprint(args)\n\nobj = function_schema.parse_args(json.dumps(args))\nobj\n
# Roundtrip test the new FunctionSchema function_schema = DataFrameFunctionSchema(pd.DataFrame) df_test = pd.DataFrame( { \"A\": [1, 2, 3], \"B\": [4, 5, 6], }, ) args = function_schema.serialize_args(df_test) print(args) obj = function_schema.parse_args(json.dumps(args)) obj 
{'columns': ['A', 'B'], 'data': [[1, 4], [2, 5], [3, 6]]}\n
Out[2]: A B 0 1 4 1 2 5 2 3 6 In\u00a0[3]: Copied! 
# Use pd.DataFrame as the return type of a prompt function\n\nimport pandas as pd\n\nfrom magentic import prompt\n\n\n@prompt(\n    \"Create a table listing the ingredients needed to cook {dish}.\"\n    \"Include a column for the quantity of each ingredient.\"\n    \"Also include a column with alergy information.\"\n)\ndef list_ingredients(dish: str) -> pd.DataFrame: ...\n\n\nlist_ingredients(\"lasagna\")\n
# Use pd.DataFrame as the return type of a prompt function import pandas as pd from magentic import prompt @prompt( \"Create a table listing the ingredients needed to cook {dish}.\" \"Include a column for the quantity of each ingredient.\" \"Also include a column with alergy information.\" ) def list_ingredients(dish: str) -> pd.DataFrame: ... list_ingredients(\"lasagna\") Out[3]: Ingredient Quantity Allergy Information 0 Lasagna noodles 16 oz Contains wheat, may contain egg and soy 1 Ground beef 1 lb Contains beef, may contain soy and gluten 2 Tomato sauce 24 oz Contains tomatoes, may contain soy and garlic 3 Mozzarella cheese 16 oz Contains milk, may contain soy 4 Ricotta cheese 15 oz Contains milk, may contain soy and eggs 5 Parmesan cheese 1 cup Contains milk, may contain soy and eggs 6 Garlic 3 cloves No known allergies 7 Onion 1 No known allergies 8 Olive oil 2 tbsp No known allergies 9 Salt 1 tsp No known allergies 10 Pepper 1/2 tsp No known allergies 11 Italian seasoning 1 tsp No known allergies 12 Sugar 1 tsp No known allergies"},{"location":"examples/registering_custom_type/#registering-a-custom-type","title":"Registering a Custom Type\u00b6","text":"

This notebook shows how to register a new type so that it can be used as the return annotation for @prompt, @promptchain, and @chatprompt. This is done by creating a new FunctionSchema which defines the parameters required to create the type, and how to parse/serialize these from/to the LLM.

See https://platform.openai.com/docs/guides/function-calling for more information on function calling, which enables this.

"},{"location":"examples/vision_renaming_screenshots/","title":"Renaming Screenshots with GPT Vision","text":"In\u00a0[1]: Copied! 
# List all screenshots\n\nfrom pathlib import Path\n\npath_desktop = Path.home() / \"Desktop\"\nscreenshot_paths = list(path_desktop.glob(\"Screenshot*.png\"))\n\nfor screenshot_path in screenshot_paths:\n    print(screenshot_path.name)\n
# List all screenshots from pathlib import Path path_desktop = Path.home() / \"Desktop\" screenshot_paths = list(path_desktop.glob(\"Screenshot*.png\")) for screenshot_path in screenshot_paths: print(screenshot_path.name) 
Screenshot 2024-04-20 at 10.49.08\u202fPM Small.png\nScreenshot 2024-04-20 at 10.50.04\u202fPM Small.png\nScreenshot 2024-04-20 at 10.50.57\u202fPM Small.png\n
In\u00a0[2]: Copied! 
# Display the first screenshot\n\nfrom IPython.display import Image, display\n\n\ndef diplay_image(image_path):\n    display(Image(data=image_path.read_bytes(), width=400))\n\n\ndiplay_image(screenshot_paths[0])\n
# Display the first screenshot from IPython.display import Image, display def diplay_image(image_path): display(Image(data=image_path.read_bytes(), width=400)) diplay_image(screenshot_paths[0]) In\u00a0[3]: Copied! 
# Define desired output for each screenshot\n# Include a description field to allow the LLM to think before naming the file\n\nfrom pydantic import BaseModel, Field\n\n\nclass ScreenshotDetails(BaseModel):\n    description: str = Field(\n        description=\"A brief description of the screenshot, including details that will be useful for naming it.\"\n    )\n    filename: str = Field(\n        description=\"An appropriate file name for this image, excluding the file extension.\"\n    )\n
# Define desired output for each screenshot # Include a description field to allow the LLM to think before naming the file from pydantic import BaseModel, Field class ScreenshotDetails(BaseModel): description: str = Field( description=\"A brief description of the screenshot, including details that will be useful for naming it.\" ) filename: str = Field( description=\"An appropriate file name for this image, excluding the file extension.\" ) In\u00a0[4]: Copied! 
# Create a prompt-function to return details given an image\n\nfrom magentic import OpenaiChatModel, Placeholder, UserMessage, chatprompt\nfrom magentic.vision import UserImageMessage\n\n\n@chatprompt(\n    UserMessage(\"Describe the screenshot, then provide a suitable file name.\"),\n    UserImageMessage(Placeholder(bytes, \"image\")),\n    model=OpenaiChatModel(\"gpt-4-turbo\"),\n)\ndef describe_image(image: bytes) -> ScreenshotDetails: ...\n
# Create a prompt-function to return details given an image from magentic import OpenaiChatModel, Placeholder, UserMessage, chatprompt from magentic.vision import UserImageMessage @chatprompt( UserMessage(\"Describe the screenshot, then provide a suitable file name.\"), UserImageMessage(Placeholder(bytes, \"image\")), model=OpenaiChatModel(\"gpt-4-turbo\"), ) def describe_image(image: bytes) -> ScreenshotDetails: ... In\u00a0[5]: Copied! 
# Rename all screenshots using the prompt-function\n\nfor path_screenshot in path_desktop.glob(\"Screenshot*.png\"):\n    print(path_screenshot.name)\n    diplay_image(path_screenshot)\n\n    image_bytes = path_screenshot.read_bytes()\n    image_details = describe_image(image_bytes)\n    print(image_details.description)\n\n    new_path = path_screenshot.with_stem(image_details.filename)\n    path_screenshot.rename(new_path)\n    print(\"\\nRenamed to:\", new_path.name)\n    print(\"\\n\\n---\\n\\n\")\n
# Rename all screenshots using the prompt-function for path_screenshot in path_desktop.glob(\"Screenshot*.png\"): print(path_screenshot.name) diplay_image(path_screenshot) image_bytes = path_screenshot.read_bytes() image_details = describe_image(image_bytes) print(image_details.description) new_path = path_screenshot.with_stem(image_details.filename) path_screenshot.rename(new_path) print(\"\\nRenamed to:\", new_path.name) print(\"\\n\\n---\\n\\n\") 
Screenshot 2024-04-20 at 10.49.08\u202fPM Small.png\n
The image shows the face of a white alpaca looking directly at the camera. The alpaca has a unique and stylish mohawk-like hairstyle. The background is blurred with a hint of green, suggesting an outdoor setting, likely a field.\n\nRenamed to: stylish-alpaca-face.png\n\n\n---\n\n\nScreenshot 2024-04-20 at 10.50.04\u202fPM Small.png\n
A close-up image of a vibrant green snake coiled around a tree branch. The snake features a beautiful pattern of yellow spots and has a focused gaze. The background is softly blurred, emphasizing the snake in the foreground.\n\nRenamed to: green_snake_coiled_on_branch.png\n\n\n---\n\n\nScreenshot 2024-04-20 at 10.50.57\u202fPM Small.png\n
The image displays a close-up view of a serving of lasagna on a white plate. The lasagna appears richly layered with melted cheese on top and a golden-brown crust, suggesting it is freshly baked and possibly contains a meaty sauce between the pasta sheets.\n\nRenamed to: close_up_lasagna_on_plate.png\n\n\n---\n\n\n
In\u00a0[\u00a0]: Copied! 
\n
"},{"location":"examples/vision_renaming_screenshots/#renaming-screenshots-with-gpt-vision","title":"Renaming Screenshots with GPT Vision\u00b6","text":"

This notebook demonstrates how to use the vision capabilites of gpt-4-turbo in combination with magentic's structured outputs to rename all those screenshots cluttering your desktop.

"}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Overview","text":"

Seamlessly integrate Large Language Models into Python code. Use the @prompt and @chatprompt decorators to create functions that return structured output from an LLM. Combine LLM queries and tool use with traditional Python code to build complex agentic systems.

"},{"location":"#features","title":"Features","text":"
  • Structured Outputs using pydantic models and built-in python types.
  • Streaming of structured outputs and function calls, to use them while being generated.
  • LLM-Assisted Retries to improve LLM adherence to complex output schemas.
  • Observability using OpenTelemetry, with native Pydantic Logfire integration.
  • Type Annotations to work nicely with linters and IDEs.
  • Configuration options for multiple LLM providers including OpenAI, Anthropic, and Ollama.
  • Many more features: Chat Prompting, Parallel Function Calling, Vision, Formatting, Asyncio...
"},{"location":"#installation","title":"Installation","text":"
pip install magentic\n

or using uv

uv add magentic\n

Configure your OpenAI API key by setting the OPENAI_API_KEY environment variable. To configure a different LLM provider see Configuration for more.

"},{"location":"#usage","title":"Usage","text":""},{"location":"#prompt","title":"@prompt","text":"

The @prompt decorator allows you to define a template for a Large Language Model (LLM) prompt as a Python function. When this function is called, the arguments are inserted into the template, then this prompt is sent to an LLM which generates the function output.

from magentic import prompt\n\n\n@prompt('Add more \"dude\"ness to: {phrase}')\ndef dudeify(phrase: str) -> str: ...  # No function body as this is never executed\n\n\ndudeify(\"Hello, how are you?\")\n# \"Hey, dude! What's up? How's it going, my man?\"\n

The @prompt decorator will respect the return type annotation of the decorated function. This can be any type supported by pydantic including a pydantic model.

from magentic import prompt\nfrom pydantic import BaseModel\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\n@prompt(\"Create a Superhero named {name}.\")\ndef create_superhero(name: str) -> Superhero: ...\n\n\ncreate_superhero(\"Garden Man\")\n# Superhero(name='Garden Man', age=30, power='Control over plants', enemies=['Pollution Man', 'Concrete Woman'])\n

See Structured Outputs for more.

"},{"location":"#chatprompt","title":"@chatprompt","text":"

The @chatprompt decorator works just like @prompt but allows you to pass chat messages as a template rather than a single text prompt. This can be used to provide a system message or for few-shot prompting where you provide example responses to guide the model's output. Format fields denoted by curly braces {example} will be filled in all messages (except FunctionResultMessage).

from magentic import chatprompt, AssistantMessage, SystemMessage, UserMessage\nfrom pydantic import BaseModel\n\n\nclass Quote(BaseModel):\n    quote: str\n    character: str\n\n\n@chatprompt(\n    SystemMessage(\"You are a movie buff.\"),\n    UserMessage(\"What is your favorite quote from Harry Potter?\"),\n    AssistantMessage(\n        Quote(\n            quote=\"It does not do to dwell on dreams and forget to live.\",\n            character=\"Albus Dumbledore\",\n        )\n    ),\n    UserMessage(\"What is your favorite quote from {movie}?\"),\n)\ndef get_movie_quote(movie: str) -> Quote: ...\n\n\nget_movie_quote(\"Iron Man\")\n# Quote(quote='I am Iron Man.', character='Tony Stark')\n

See Chat Prompting for more.

"},{"location":"#functioncall","title":"FunctionCall","text":"

An LLM can also decide to call functions. In this case the @prompt-decorated function returns a FunctionCall object which can be called to execute the function using the arguments provided by the LLM.

from typing import Literal\n\nfrom magentic import prompt, FunctionCall\n\n\ndef search_twitter(query: str, category: Literal[\"latest\", \"people\"]) -> str:\n    \"\"\"Searches Twitter for a query.\"\"\"\n    print(f\"Searching Twitter for {query!r} in category {category!r}\")\n    return \"<twitter results>\"\n\n\ndef search_youtube(query: str, channel: str = \"all\") -> str:\n    \"\"\"Searches YouTube for a query.\"\"\"\n    print(f\"Searching YouTube for {query!r} in channel {channel!r}\")\n    return \"<youtube results>\"\n\n\n@prompt(\n    \"Use the appropriate search function to answer: {question}\",\n    functions=[search_twitter, search_youtube],\n)\ndef perform_search(question: str) -> FunctionCall[str]: ...\n\n\noutput = perform_search(\"What is the latest news on LLMs?\")\nprint(output)\n# > FunctionCall(<function search_twitter at 0x10c367d00>, 'LLMs', 'latest')\noutput()\n# > Searching Twitter for 'Large Language Models news' in category 'latest'\n# '<twitter results>'\n

See Function Calling for more.

"},{"location":"#prompt_chain","title":"@prompt_chain","text":"

Sometimes the LLM requires making one or more function calls to generate a final answer. The @prompt_chain decorator will resolve FunctionCall objects automatically and pass the output back to the LLM to continue until the final answer is reached.

In the following example, when describe_weather is called the LLM first calls the get_current_weather function, then uses the result of this to formulate its final answer which gets returned.

from magentic import prompt_chain\n\n\ndef get_current_weather(location, unit=\"fahrenheit\"):\n    \"\"\"Get the current weather in a given location\"\"\"\n    # Pretend to query an API\n    return {\n        \"location\": location,\n        \"temperature\": \"72\",\n        \"unit\": unit,\n        \"forecast\": [\"sunny\", \"windy\"],\n    }\n\n\n@prompt_chain(\n    \"What's the weather like in {city}?\",\n    functions=[get_current_weather],\n)\ndef describe_weather(city: str) -> str: ...\n\n\ndescribe_weather(\"Boston\")\n# 'The current weather in Boston is 72\u00b0F and it is sunny and windy.'\n

LLM-powered functions created using @prompt, @chatprompt and @prompt_chain can be supplied as functions to other @prompt/@prompt_chain decorators, just like regular python functions. This enables increasingly complex LLM-powered functionality, while allowing individual components to be tested and improved in isolation.

"},{"location":"asyncio/","title":"Asyncio","text":"

Asynchronous functions / coroutines can be used to concurrently query the LLM. This can greatly increase the overall speed of generation, and also allow other asynchronous code to run while waiting on LLM output. In the below example, the LLM generates a description for each US president while it is waiting on the next one in the list. Measuring the characters generated per second shows that this example achieves a 7x speedup over serial processing.

import asyncio\nfrom time import time\nfrom typing import AsyncIterable\n\nfrom magentic import prompt\n\n\n@prompt(\"List ten presidents of the United States\")\nasync def iter_presidents() -> AsyncIterable[str]: ...\n\n\n@prompt(\"Tell me more about {topic}\")\nasync def tell_me_more_about(topic: str) -> str: ...\n\n\n# For each president listed, generate a description concurrently\nstart_time = time()\ntasks = []\nasync for president in await iter_presidents():\n    # Use asyncio.create_task to schedule the coroutine for execution before awaiting it\n    # This way descriptions will start being generated while the list of presidents is still being generated\n    task = asyncio.create_task(tell_me_more_about(president))\n    tasks.append(task)\n\ndescriptions = await asyncio.gather(*tasks)\n\n# Measure the characters per second\ntotal_chars = sum(len(desc) for desc in descriptions)\ntime_elapsed = time() - start_time\nprint(total_chars, time_elapsed, total_chars / time_elapsed)\n# 24575 28.70 856.07\n\n\n# Measure the characters per second to describe a single president\nstart_time = time()\nout = await tell_me_more_about(\"George Washington\")\ntime_elapsed = time() - start_time\nprint(len(out), time_elapsed, len(out) / time_elapsed)\n# 2206 18.72 117.78\n
"},{"location":"chat-prompting/","title":"Chat Prompting","text":""},{"location":"chat-prompting/#chatprompt","title":"@chatprompt","text":"

The @chatprompt decorator works just like @prompt but allows you to pass chat messages as a template rather than a single text prompt. This can be used to provide a system message or for few-shot prompting where you provide example responses to guide the model's output. Format fields denoted by curly braces {example} will be filled in all messages (except FunctionResultMessage).

from magentic import chatprompt, AssistantMessage, SystemMessage, UserMessage\nfrom pydantic import BaseModel\n\n\nclass Quote(BaseModel):\n    quote: str\n    character: str\n\n\n@chatprompt(\n    SystemMessage(\"You are a movie buff.\"),\n    UserMessage(\"What is your favorite quote from Harry Potter?\"),\n    AssistantMessage(\n        Quote(\n            quote=\"It does not do to dwell on dreams and forget to live.\",\n            character=\"Albus Dumbledore\",\n        )\n    ),\n    UserMessage(\"What is your favorite quote from {movie}?\"),\n)\ndef get_movie_quote(movie: str) -> Quote: ...\n\n\nget_movie_quote(\"Iron Man\")\n# Quote(quote='I am Iron Man.', character='Tony Stark')\n
"},{"location":"chat-prompting/#escape_braces","title":"escape_braces","text":"

To prevent curly braces from being interpreted as format fields, use the escape_braces function to escape them in strings.

from magentic.chatprompt import escape_braces\n\nstring_with_braces = \"Curly braces like {example} will be filled in!\"\nescaped_string = escape_braces(string_with_braces)\n# 'Curly braces {{example}} will be filled in!'\nescaped_string.format(example=\"test\")\n# 'Curly braces {example} will be filled in!'\n
"},{"location":"chat-prompting/#placeholder","title":"Placeholder","text":"

The Placeholder class enables templating of AssistantMessage content within the @chatprompt decorator. This allows dynamic changing of the messages used to prompt the model based on the arguments provided when the function is called.

from magentic import chatprompt, AssistantMessage, Placeholder, UserMessage\nfrom pydantic import BaseModel\n\n\nclass Quote(BaseModel):\n    quote: str\n    character: str\n\n\n@chatprompt(\n    UserMessage(\"Tell me a quote from {movie}\"),\n    AssistantMessage(Placeholder(Quote, \"quote\")),\n    UserMessage(\"What is a similar quote from the same movie?\"),\n)\ndef get_similar_quote(movie: str, quote: Quote) -> Quote: ...\n\n\nget_similar_quote(\n    movie=\"Star Wars\",\n    quote=Quote(quote=\"I am your father\", character=\"Darth Vader\"),\n)\n# Quote(quote='The Force will be with you, always.', character='Obi-Wan Kenobi')\n

Placeholder can also be utilized in the format method of custom Message subclasses to provide an explicit way of inserting values from the function arguments. For example, see UserImageMessage in (TODO: link to GPT-vision page).

"},{"location":"chat-prompting/#functioncall","title":"FunctionCall","text":"

The content of an AssistantMessage can be a FunctionCall. This can be used to demonstrate to the LLM when/how it should call a function.

from magentic import (\n    chatprompt,\n    AssistantMessage,\n    FunctionCall,\n    UserMessage,\n    SystemMessage,\n)\n\n\ndef change_music_volume(increment: int) -> int:\n    \"\"\"Change music volume level. Min 1, max 10.\"\"\"\n    print(f\"Music volume change: {increment}\")\n\n\ndef order_food(food: str, amount: int):\n    \"\"\"Order food.\"\"\"\n    print(f\"Ordered {amount} {food}\")\n\n\n@chatprompt(\n    SystemMessage(\n        \"You are hosting a party and must keep the guests happy.\"\n        \"Call functions as needed. Do not respond directly.\"\n    ),\n    UserMessage(\"It's pretty loud in here!\"),\n    AssistantMessage(FunctionCall(change_music_volume, -2)),\n    UserMessage(\"{request}\"),\n    functions=[change_music_volume, order_food],\n)\ndef adjust_for_guest(request: str) -> FunctionCall[None]: ...\n\n\nfunc = adjust_for_guest(\"Do you have any more food?\")\nfunc()\n# Ordered 3 pizza\n
"},{"location":"chat-prompting/#functionresultmessage","title":"FunctionResultMessage","text":"

To include the result of calling the function in the messages use a FunctionResultMessage. This takes a FunctionCall instance as its second argument. The same FunctionCall instance must be passed to an AssistantMessage and the corresponding FunctionResultMessage so that the result can be correctly linked back to the function call that created it.

from magentic import (\n    chatprompt,\n    AssistantMessage,\n    FunctionCall,\n    FunctionResultMessage,\n    UserMessage,\n)\n\n\ndef plus(a: int, b: int) -> int:\n    return a + b\n\n\nplus_1_2 = FunctionCall(plus, 1, 2)\n\n\n@chatprompt(\n    UserMessage(\"Use the plus function to add 1 and 2.\"),\n    AssistantMessage(plus_1_2),\n    FunctionResultMessage(3, plus_1_2),\n    UserMessage(\"Now add 4 to the result.\"),\n    functions=[plus],\n)\ndef do_math() -> FunctionCall[int]: ...\n\n\ndo_math()\n# FunctionCall(<function plus at 0x10a0829e0>, 3, 4)\n
"},{"location":"chat-prompting/#anymessage","title":"AnyMessage","text":"

The AnyMessage type can be used for (de)serialization of Message objects, or as a return type in prompt-functions. This allows you to create prompt-functions to do things like summarize a chat history into fewer messages, or even to create a set of messages that you can use in a chatprompt-function.

from magentic import AnyMessage, prompt\n\n\n@prompt(\"Create an example of few-shot prompting for a chatbot\")\ndef make_few_shot_prompt() -> list[AnyMessage]: ...\n\n\nmake_few_shot_prompt()\n# [SystemMessage('You are a helpful and knowledgeable assistant. You answer questions promptly and accurately. Always be polite and concise.'),\n#  UserMessage('What\u2019s the weather like today?'),\n#  AssistantMessage[Any]('The weather today is sunny with a high of 75\u00b0F (24\u00b0C) and a low of 55\u00b0F (13\u00b0C). No chance of rain.'),\n#  UserMessage('Can you explain the theory of relativity in simple terms?'),\n#  AssistantMessage[Any]('Sure! The theory of relativity, developed by Albert Einstein, has two main parts: Special Relativity and General Relativity. Special Relativity is about how time and space are linked for objects moving at a consistent speed in a straight line. It shows that time can slow down or speed up depending on how fast you are moving compared to something else. General Relativity adds gravity into the mix and shows that massive objects cause space to bend and warp, which we feel as gravity.'),\n#  UserMessage('How do I bake a chocolate cake?'),\n#  AssistantMessage[Any](\"Here's a simple recipe for a chocolate cake:\\n\\nIngredients:\\n- 1 and 3/4 cups all-purpose flour\\n- 1 and 1/2 cups granulated sugar\\n- 3/4 cup cocoa powder\\n- 1 and 1/2 teaspoons baking powder\\n- 1 and 1/2 teaspoons baking soda\\n- 1 teaspoon salt\\n- 2 large eggs\\n- 1 cup whole milk\\n- 1/2 cup vegetable oil\\n- 2 teaspoons vanilla extract\\n- 1 cup boiling water\\n\\nInstructions:\\n1. Preheat your oven to 350\u00b0F (175\u00b0C). Grease and flour two 9-inch round baking pans.\\n2. In a large bowl, whisk together the flour, sugar, cocoa powder, baking powder, baking soda, and salt.\\n3. Add the eggs, milk, oil, and vanilla. Beat on medium speed for 2 minutes.\\n4. Stir in the boiling water (batter will be thin).\\n5. Pour the batter evenly into the prepared pans.\\n6. Bake for 30 to 35 minutes or until a toothpick inserted into the center comes out clean.\\n7. Cool the cakes in the pans for 10 minutes, then remove them from the pans and cool completely on a wire rack.\\n8. Frost with your favorite chocolate frosting and enjoy!\")]\n

For (de)serialization, check out TypeAdapter from pydantic. See more on the pydantic Type Adapter docs page.

from magentic import AnyMessage\nfrom pydantic import TypeAdapter\n\n\nmessages = [\n    {\"role\": \"system\", \"content\": \"Hello\"},\n    {\"role\": \"user\", \"content\": \"Hello\"},\n    {\"role\": \"assistant\", \"content\": \"Hello\"},\n    {\"role\": \"tool\", \"content\": 3, \"tool_call_id\": \"unique_id\"},\n]\nTypeAdapter(list[AnyMessage]).validate_python(messages)\n# [SystemMessage('Hello'),\n#  UserMessage('Hello'),\n#  AssistantMessage[Any]('Hello'),\n#  ToolResultMessage[Any](3, self.tool_call_id='unique_id')]\n
"},{"location":"configuration/","title":"LLM Configuration","text":""},{"location":"configuration/#backends","title":"Backends","text":"

Magentic supports multiple LLM providers or \"backends\". This roughly refers to which Python package is used to interact with the LLM API. The following backends are supported.

"},{"location":"configuration/#openai","title":"OpenAI","text":"

The default backend, using the openai Python package and supports all features of magentic.

No additional installation is required. Just import the OpenaiChatModel class from magentic.

from magentic import OpenaiChatModel\n\nmodel = OpenaiChatModel(\"gpt-4o\")\n
"},{"location":"configuration/#ollama-via-openai","title":"Ollama via OpenAI","text":"

Ollama supports an OpenAI-compatible API, which allows you to use Ollama models via the OpenAI backend.

First, install ollama from ollama.com. Then, pull the model you want to use.

ollama pull llama3.2\n

Then, specify the model name and base_url when creating the OpenaiChatModel instance.

from magentic import OpenaiChatModel\n\nmodel = OpenaiChatModel(\"llama3.2\", base_url=\"http://localhost:11434/v1/\")\n
"},{"location":"configuration/#other-openai-compatible-apis","title":"Other OpenAI-compatible APIs","text":"

When using the openai backend, setting the MAGENTIC_OPENAI_BASE_URL environment variable or using OpenaiChatModel(..., base_url=\"http://localhost:8080\") in code allows you to use magentic with any OpenAI-compatible API e.g. Azure OpenAI Service, LiteLLM OpenAI Proxy Server, LocalAI. Note that if the API does not support tool calls then you will not be able to create prompt-functions that return Python objects, but other features of magentic will still work.

To use Azure with the openai backend you will need to set the MAGENTIC_OPENAI_API_TYPE environment variable to \"azure\" or use OpenaiChatModel(..., api_type=\"azure\"), and also set the environment variables needed by the openai package to access Azure. See https://github.com/openai/openai-python#microsoft-azure-openai

"},{"location":"configuration/#anthropic","title":"Anthropic","text":"

This uses the anthropic Python package and supports all features of magentic.

Install the magentic package with the anthropic extra, or install the anthropic package directly.

pip install \"magentic[anthropic]\"\n

Then import the AnthropicChatModel class.

from magentic.chat_model.anthropic_chat_model import AnthropicChatModel\n\nmodel = AnthropicChatModel(\"claude-3-5-sonnet-latest\")\n
"},{"location":"configuration/#litellm","title":"LiteLLM","text":"

This uses the litellm Python package to enable querying LLMs from many different providers. Note: some models may not support all features of magentic e.g. function calling/structured output and streaming.

Install the magentic package with the litellm extra, or install the litellm package directly.

pip install \"magentic[litellm]\"\n

Then import the LitellmChatModel class.

from magentic.chat_model.litellm_chat_model import LitellmChatModel\n\nmodel = LitellmChatModel(\"gpt-4o\")\n
"},{"location":"configuration/#mistral","title":"Mistral","text":"

This uses the openai Python package with some small modifications to make the API queries compatible with the Mistral API. It supports all features of magentic. However tool calls (including structured outputs) are not streamed so are received all at once.

Note: a future version of magentic might switch to using the mistral Python package.

No additional installation is required. Just import the MistralChatModel class.

from magentic.chat_model.mistral_chat_model import MistralChatModel\n\nmodel = MistralChatModel(\"mistral-large-latest\")\n
"},{"location":"configuration/#configure-a-backend","title":"Configure a Backend","text":"

The default ChatModel used by magentic (in @prompt, @chatprompt, etc.) can be configured in several ways. When a prompt-function or chatprompt-function is called, the ChatModel to use follows this order of preference

  1. The ChatModel instance provided as the model argument to the magentic decorator
  2. The current chat model context, created using with MyChatModel:
  3. The global ChatModel created from environment variables and the default settings in src/magentic/settings.py

The following code snippet demonstrates this behavior:

from magentic import OpenaiChatModel, prompt\nfrom magentic.chat_model.anthropic_chat_model import AnthropicChatModel\n\n\n@prompt(\"Say hello\")\ndef say_hello() -> str: ...\n\n\n@prompt(\n    \"Say hello\",\n    model=AnthropicChatModel(\"claude-3-5-sonnet-latest\"),\n)\ndef say_hello_anthropic() -> str: ...\n\n\nsay_hello()  # Uses env vars or default settings\n\nwith OpenaiChatModel(\"gpt-4o-mini\", temperature=1):\n    say_hello()  # Uses openai with gpt-4o-mini and temperature=1 due to context manager\n    say_hello_anthropic()  # Uses Anthropic claude-3-5-sonnet-latest because explicitly configured\n

The following environment variables can be set.

Environment Variable Description Example MAGENTIC_BACKEND The package to use as the LLM backend anthropic / openai / litellm MAGENTIC_ANTHROPIC_MODEL Anthropic model claude-3-haiku-20240307 MAGENTIC_ANTHROPIC_API_KEY Anthropic API key to be used by magentic sk-... MAGENTIC_ANTHROPIC_BASE_URL Base URL for an Anthropic-compatible API http://localhost:8080 MAGENTIC_ANTHROPIC_MAX_TOKENS Max number of generated tokens 1024 MAGENTIC_ANTHROPIC_TEMPERATURE Temperature 0.5 MAGENTIC_LITELLM_MODEL LiteLLM model claude-2 MAGENTIC_LITELLM_API_BASE The base url to query http://localhost:11434 MAGENTIC_LITELLM_MAX_TOKENS LiteLLM max number of generated tokens 1024 MAGENTIC_LITELLM_TEMPERATURE LiteLLM temperature 0.5 MAGENTIC_MISTRAL_MODEL Mistral model mistral-large-latest MAGENTIC_MISTRAL_API_KEY Mistral API key to be used by magentic XEG... MAGENTIC_MISTRAL_BASE_URL Base URL for an Mistral-compatible API http://localhost:8080 MAGENTIC_MISTRAL_MAX_TOKENS Max number of generated tokens 1024 MAGENTIC_MISTRAL_SEED Seed for deterministic sampling 42 MAGENTIC_MISTRAL_TEMPERATURE Temperature 0.5 MAGENTIC_OPENAI_MODEL OpenAI model gpt-4 MAGENTIC_OPENAI_API_KEY OpenAI API key to be used by magentic sk-... MAGENTIC_OPENAI_API_TYPE Allowed options: \"openai\", \"azure\" azure MAGENTIC_OPENAI_BASE_URL Base URL for an OpenAI-compatible API http://localhost:8080 MAGENTIC_OPENAI_MAX_TOKENS OpenAI max number of generated tokens 1024 MAGENTIC_OPENAI_SEED Seed for deterministic sampling 42 MAGENTIC_OPENAI_TEMPERATURE OpenAI temperature 0.5"},{"location":"formatting/","title":"Formatting","text":""},{"location":"formatting/#the-format-method","title":"The format Method","text":"

Functions created using magentic decorators expose a format method that accepts the same parameters as the function itself but returns the completed prompt that will be sent to the model. For @prompt this method returns a string, and for @chatprompt it returns a list of Message objects. The format method can be used to test that the final prompt created by a magentic function is formatted as expected.

from magentic import prompt\n\n\n@prompt(\"Write a short poem about {topic}.\")\ndef create_poem(topic: str) -> str: ...\n\n\ncreate_poem.format(\"fruit\")\n# 'Write a short poem about fruit.'\n
"},{"location":"formatting/#classes-for-formatting","title":"Classes for Formatting","text":"

By default, when a list is used in a prompt template string it is formatted using its Python representation.

from magentic import prompt\nfrom magentic.formatting import BulletedList\n\n\n@prompt(\"Continue the list:\\n{items}\")\ndef get_next_items(items: list[str]) -> list[str]: ...\n\n\nitems = [\"apple\", \"banana\", \"cherry\"]\nprint(get_next_items.format(items=items))\n# Continue the list:\n# ['apple', 'banana', 'cherry']\n

However, the LLM might respond better to a prompt in which the list is formatted more clearly or the items are numbered. The BulletedList, NumberedList, BulletedDict and NumberedDict classes are provided to enable this.

For example, to modify the above prompt to contain a numbered list of the items, the NumberedList class can be used. This behaves exactly like a regular Python list except for how it appears when inserted into a formatted string. This class can also be used as the type annotation for items parameter to ensure that this prompt always contains a numbered list.

from magentic import prompt\nfrom magentic.formatting import NumberedList\n\n\n@prompt(\"Continue the list:\\n{items}\")\ndef get_next_items(items: NumberedList[str]) -> list[str]: ...\n\n\nitems = NumberedList([\"apple\", \"banana\", \"cherry\"])\nprint(get_next_items.format(items=items))\n# Continue the list:\n# 1. apple\n# 2. banana\n# 3. cherry\n
"},{"location":"formatting/#custom-formatting","title":"Custom Formatting","text":"

When objects are inserted into formatted strings in Python, the __format__ method is called. By defining or modifying this method you can control how an object is converted to a string in the prompt. If you own the class you can modify the __format__ method directly. Otherwise for third-party classes you will need to create a subcless.

Here's an example of how to represent a dictionary as a bulleted list.

from typing import TypeVar\n\nfrom magentic import prompt\n\nK = TypeVar(\"K\")\nV = TypeVar(\"V\")\n\n\nclass BulletedDict(dict[K, V]):\n    def __format__(self, format_spec: str) -> str:\n        # Here, you could use 'format_spec' to customize the formatting further if needed\n        return \"\\n\".join(f\"- {key}: {value}\" for key, value in self.items())\n\n\n@prompt(\"Identify the odd one out:\\n{items}\")\ndef find_odd_one_out(items: BulletedDict[str, str]) -> str: ...\n\n\nitems = BulletedDict({\"sky\": \"blue\", \"grass\": \"green\", \"sun\": \"purple\"})\nprint(find_odd_one_out.format(items))\n# Identify the odd one out:\n# - sky: blue\n# - grass: green\n# - sun: purple\n
"},{"location":"function-calling/","title":"Function Calling","text":"

For many use cases, it is useful to provide the LLM with tools that it can choose when and how to use. In magentic this is done by passing a list of Python functions to the functions argument of a magentic decorator.

If the LLM chooses to call a function, the decorated function will return a FunctionCall instance. This object can be called to execute the function with the arguments that the LLM provided.

from typing import Literal\n\nfrom magentic import prompt, FunctionCall\n\n\ndef search_twitter(query: str, category: Literal[\"latest\", \"people\"]) -> str:\n    \"\"\"Searches Twitter for a query.\"\"\"\n    print(f\"Searching Twitter for {query!r} in category {category!r}\")\n    return \"<twitter results>\"\n\n\ndef search_youtube(query: str, channel: str = \"all\") -> str:\n    \"\"\"Searches YouTube for a query.\"\"\"\n    print(f\"Searching YouTube for {query!r} in channel {channel!r}\")\n    return \"<youtube results>\"\n\n\n@prompt(\n    \"Use the appropriate search function to answer: {question}\",\n    functions=[search_twitter, search_youtube],\n)\ndef perform_search(question: str) -> FunctionCall[str]: ...\n\n\noutput = perform_search(\"What is the latest news on LLMs?\")\nprint(output)\n# > FunctionCall(<function search_twitter at 0x10c367d00>, 'LLMs', 'latest')\noutput()\n# > Searching Twitter for 'Large Language Models news' in category 'latest'\n# '<twitter results>'\n
"},{"location":"function-calling/#functioncall","title":"FunctionCall","text":"

A FunctionCall combines a function with a set of arguments, ready to be called with no additional inputs required. In magentic, each time the LLM chooses to invoke a function a FunctionCall instance is returned. This allows the chosen function and supplied arguments to be validated or logged before the function is executed.

from magentic import FunctionCall\n\n\ndef plus(a: int, b: int) -> int:\n    return a + b\n\n\nplus_1_2 = FunctionCall(plus, 1, b=2)\nprint(plus_1_2.function)\n# > <function plus at 0x10c39cd30>\nprint(plus_1_2.arguments)\n# > {'a': 1, 'b': 2}\nplus_1_2()\n# 3\n
"},{"location":"function-calling/#prompt_chain","title":"@prompt_chain","text":"

In some cases, you need the model to perform multiple function calls to reach a final answer. The @prompt_chain decorator will execute function calls automatically, append the result to the list of messages, and query the LLM again until a final answer is reached.

In the following example, when describe_weather is called the LLM first calls the get_current_weather function, then uses the result of this to formulate its final answer which gets returned.

from magentic import prompt_chain\n\n\ndef get_current_weather(location, unit=\"fahrenheit\"):\n    \"\"\"Get the current weather in a given location\"\"\"\n    # Pretend to query an API\n    return {\n        \"location\": location,\n        \"temperature\": \"72\",\n        \"unit\": unit,\n        \"forecast\": [\"sunny\", \"windy\"],\n    }\n\n\n@prompt_chain(\n    \"What's the weather like in {city}?\",\n    functions=[get_current_weather],\n)\ndef describe_weather(city: str) -> str: ...\n\n\ndescribe_weather(\"Boston\")\n# 'The current weather in Boston is 72\u00b0F and it is sunny and windy.'\n

LLM-powered functions created using @prompt, @chatprompt and @prompt_chain can be supplied as functions to other @prompt/@prompt_chain decorators, just like regular python functions!

"},{"location":"function-calling/#parallelfunctioncall","title":"ParallelFunctionCall","text":"

The most recent LLMs support \"parallel function calling\". This allows the model to call multiple functions at once. These functions can be executed concurrently, avoiding having to make several serial queries to the model.

You can use ParallelFunctionCall (and AsyncParallelFunctionCall) as a return annotation to indicate that you expect the LLM to make one or more function calls. The returned ParallelFunctionCall is a container of FunctionCall instances. When called, it returns a tuple of their results.

from typing import Literal\n\nfrom magentic import prompt, ParallelFunctionCall\n\n\ndef search_twitter(query: str, category: Literal[\"latest\", \"people\"]) -> str:\n    \"\"\"Searches Twitter for a query.\"\"\"\n    print(f\"Searching Twitter for {query!r} in category {category!r}\")\n    return \"<twitter results>\"\n\n\ndef search_youtube(query: str, channel: str = \"all\") -> str:\n    \"\"\"Searches YouTube for a query.\"\"\"\n    print(f\"Searching YouTube for {query!r} in channel {channel!r}\")\n    return \"<youtube results>\"\n\n\n@prompt(\n    \"Use the appropriate search functions to answer: {question}\",\n    functions=[search_twitter, search_youtube],\n)\ndef perform_search(question: str) -> ParallelFunctionCall[str]: ...\n\n\noutput = perform_search(\"What is the latest news on LLMs?\")\nprint(list(output))\n# > [FunctionCall(<function search_twitter at 0x10c39f760>, 'LLMs', 'latest'),\n#    FunctionCall(<function search_youtube at 0x10c39f7f0>, 'LLMs')]\noutput()\n# > Searching Twitter for 'LLMs' in category 'latest'\n# > Searching YouTube for 'LLMs' in channel 'all'\n# ('<twitter results>', '<youtube results>')\n
"},{"location":"function-calling/#parallelfunctioncall-with-chatprompt","title":"ParallelFunctionCall with @chatprompt","text":"

As with FunctionCall and Pydantic/Python objects, ParallelFunctionCall can be used with @chatprompt for few-shot prompting. In other words, to demonstrate to the LLM how/when it should use functions.

from magentic import (\n    chatprompt,\n    AssistantMessage,\n    FunctionCall,\n    FunctionResultMessage,\n    ParallelFunctionCall,\n    UserMessage,\n)\n\n\ndef plus(a: int, b: int) -> int:\n    return a + b\n\n\ndef minus(a: int, b: int) -> int:\n    return a - b\n\n\nplus_1_2 = FunctionCall(plus, 1, 2)\nminus_2_1 = FunctionCall(minus, 2, 1)\n\n\n@chatprompt(\n    UserMessage(\n        \"Sum 1 and 2. Also subtract 1 from 2.\",\n    ),\n    AssistantMessage(ParallelFunctionCall([plus_1_2, minus_2_1])),\n    FunctionResultMessage(3, plus_1_2),\n    FunctionResultMessage(1, minus_2_1),\n    UserMessage(\"Now add 4 to both results.\"),\n    functions=[plus, minus],\n)\ndef do_math() -> ParallelFunctionCall[int]: ...\n\n\noutput = do_math()\nprint(list(output))\n# > [FunctionCall(<function plus at 0x10c3584c0>, 3, 4),\n#    FunctionCall(<function plus at 0x10c3584c0>, 1, 4)]\noutput()\n# (7, 5)\n
"},{"location":"function-calling/#annotated-parameters","title":"Annotated Parameters","text":"

Like with BaseModel, you can use pydantic's Field to provide additional information for individual function parameters, such as a description. Here's how you could document for the model that the temperature parameter of the activate_oven function is measured in Fahrenheit and should be less than 500.

from typing import Annotated, Literal\n\nfrom pydantic import Field\n\n\ndef activate_oven(\n    temperature: Annotated[int, Field(description=\"Temp in Fahrenheit\", lt=500)],\n    mode: Literal[\"broil\", \"bake\", \"roast\"],\n) -> str:\n    \"\"\"Turn the oven on with the provided settings.\"\"\"\n    return f\"Preheating to {temperature} F with mode {mode}\"\n
"},{"location":"function-calling/#configdict","title":"ConfigDict","text":"

Also like with BaseModel, pydantic's (or magentic's) ConfigDict can be used with functions to configure behavior. Under the hood, the function gets converted to a pydantic model, with every function parameter becoming a field on that model. See the Structured Outputs docs page for more information including the list of configuration options added by magentic.

from typing import Annotated, Literal\n\nfrom magentic import ConfigDict, with_config\nfrom pydantic import Field\n\n\n@with_config(ConfigDict(openai_strict=True))\ndef activate_oven(\n    temperature: Annotated[int, Field(description=\"Temp in Fahrenheit\", lt=500)],\n    mode: Literal[\"broil\", \"bake\", \"roast\"],\n) -> str:\n    \"\"\"Turn the oven on with the provided settings.\"\"\"\n    return f\"Preheating to {temperature} F with mode {mode}\"\n
"},{"location":"logging-and-tracing/","title":"Logging and Tracing","text":"

magentic is instrumented for logging and tracing using Pydantic Logfire. This also makes it compatible with OpenTelemetry.

"},{"location":"logging-and-tracing/#logging-to-stdout","title":"Logging to stdout","text":"

To print magentic spans and logs to stdout, first install the logfire Python package.

pip install logfire\n

Then configure this as needed. See the Pydantic Logfire Integrations docs for the full list of integrations and how to configure these.

import logfire\n\nlogfire.configure(send_to_logfire=False)\nlogfire.instrument_openai()  # optional, to trace OpenAI API calls\n# logfire.instrument_anthropic()  # optional, to trace Anthropic API calls\n

Now when running magentic code, all spans and logs will be printed.

23:02:34.197 Calling prompt-chain describe_weather\n23:02:34.221   Chat Completion with 'gpt-4o' [LLM]\n23:02:35.364   streaming response from 'gpt-4o' took 0.05s [LLM]\n23:02:35.365   Executing function call get_current_weather\n23:02:35.399   Chat Completion with 'gpt-4o' [LLM]\n23:02:35.992   streaming response from 'gpt-4o' took 0.12s [LLM]\n
"},{"location":"logging-and-tracing/#using-pydantic-logfire","title":"Using Pydantic Logfire","text":"

Create a Pydantic Logfire account following their First Steps Guide.

After authenticating locally and creating a project, configure logfire

import logfire\n\nlogfire.configure()  # note: `send_to_logfire` removed. This defaults to True\nlogfire.instrument_openai()  # optional, to trace OpenAI API calls\n# logfire.instrument_anthropic()  # optional, to trace Anthropic API calls\n

Now calls to magentic prompt-functions, prompt-chains etc. will become visible in the Logfire UI.

"},{"location":"logging-and-tracing/#configuring-opentelemetry","title":"Configuring OpenTelemetry","text":"

To enable instrumentation for use with OpenTelemetry, use the following logfire configuration

import logfire\n\nlogfire.configure(\n    send_to_logfire=False,\n    service_name=\"magentic-test\",  # optional, can be set using OTEL_SERVICE_NAME env var\n)\nlogfire.instrument_openai()  # optional, to trace OpenAI API calls\n# logfire.instrument_anthropic()  # optional, to trace Anthropic API calls\n

Now logs and traces for magentic (and OpenAI, Anthropic, ...) will be available to any OTEL tracers.

"},{"location":"logging-and-tracing/#viewing-traces-locally","title":"Viewing Traces Locally","text":"

To view traces locally you can use Jaeger.

First start the Jaeger all-in-one docker container

docker run --rm --name jaeger \\\n  -p 16686:16686 \\\n  -p 4317:4317 \\\n  -p 4318:4318 \\\n  jaegertracing/all-in-one:1.58\n

Then navigate to http://localhost:16686 to access the Jaeger UI. See the Jaeger Getting Started Guide for up-to-date instructions.

Next, install the required OpenTelemetry exporter and configure OpenTelemetry to use this to send traces to Jaeger.

pip install opentelemetry-exporter-otlp\n
from opentelemetry import trace\nfrom opentelemetry.sdk.trace.export import BatchSpanProcessor\nfrom opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter\n\ntrace.get_tracer_provider().add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))\n

Now, traces for magentic code run locally will be visible in the Jaeger UI.

"},{"location":"logging-and-tracing/#enabling-debug-logging","title":"Enabling Debug Logging","text":"

The neatest way to view the raw requests sent to LLM provider APIs is to use Logfire as described above. Another method is to enable debug logs for the LLM provider's Python package. The openai and anthropic packages use the standard library logger and expose an environment variable to set the log level. See the Logging section of the openai README or Logging section of the anthropic README for more information.

Here's an example of what the debug log contains for a simple magentic prompt-function (formatted to improve readability).

import logging\n\nfrom magentic import prompt\n\nlogging.basicConfig(level=logging.DEBUG)\n\n\ndef plus(a: int, b: int) -> int:\n    return a + b\n\n\n@prompt(\n    \"Say hello {n} times\",\n    functions=[plus],\n)\ndef say_hello(n: int) -> str: ...\n\n\nsay_hello(2)\n# ...\n# > DEBUG:openai._base_client:Request options: {\n#     \"method\": \"post\",\n#     \"url\": \"/chat/completions\",\n#     \"files\": None,\n#     \"json_data\": {\n#         \"messages\": [{\"role\": \"user\", \"content\": \"Say hello 2 times\"}],\n#         \"model\": \"gpt-3.5-turbo\",\n#         \"functions\": [\n#             {\n#                 \"name\": \"plus\",\n#                 \"parameters\": {\n#                     \"properties\": {\n#                         \"a\": {\"title\": \"A\", \"type\": \"integer\"},\n#                         \"b\": {\"title\": \"B\", \"type\": \"integer\"},\n#                     },\n#                     \"required\": [\"a\", \"b\"],\n#                     \"type\": \"object\",\n#                 },\n#             }\n#         ],\n#         \"max_tokens\": None,\n#         \"stream\": True,\n#         \"temperature\": None,\n#     },\n# }\n# ...\n
"},{"location":"retrying/","title":"Retrying","text":""},{"location":"retrying/#llm-assisted-retries","title":"LLM-Assisted Retries","text":"

Occasionally the LLM returns an output that cannot be parsed into any of the output types or function calls that were requested. Additionally, the pydantic models you define might have extra validation that is not represented by the type annotations alone. In these cases, LLM-assisted retries can be used to automatically resubmit the output as well as the associated error message back to the LLM, giving it another opportunity with more information to meet the output schema requirements.

To enable retries, simply set the max_retries parameter to a non-zero value in @prompt or @chatprompt.

In this example

  • the LLM first returns a country that is not Ireland
  • then the pydantic model validation fails with error \"Country must be Ireland\"
  • the original output as well as a message containing the error are resubmitted to the LLM
  • the LLM correctly meets the output requirement returning \"Ireland\"
from typing import Annotated\n\nfrom magentic import prompt\nfrom pydantic import AfterValidator, BaseModel\n\n\ndef assert_is_ireland(v: str) -> str:\n    if v != \"Ireland\":\n        raise ValueError(\"Country must be Ireland\")\n    return v\n\n\nclass Country(BaseModel):\n    name: Annotated[str, AfterValidator(assert_is_ireland)]\n    capital: str\n\n\n@prompt(\n    \"Return a country\",\n    max_retries=3,\n)\ndef get_country() -> Country: ...\n\n\nget_country()\n# 05:13:55.607 Calling prompt-function get_country\n# 05:13:55.622   LLM-assisted retries enabled. Max 3\n# 05:13:55.627     Chat Completion with 'gpt-4o' [LLM]\n# 05:13:56.309     streaming response from 'gpt-4o' took 0.11s [LLM]\n# 05:13:56.310     Retrying Chat Completion. Attempt 1.\n# 05:13:56.322     Chat Completion with 'gpt-4o' [LLM]\n# 05:13:57.456     streaming response from 'gpt-4o' took 0.00s [LLM]\n#\n# Country(name='Ireland', capital='Dublin')\n

LLM-Assisted retries are intended to address cases where the LLM failed to generate valid output. Errors due to LLM provider rate limiting, internet connectivity issues, or other issues that cannot be solved by reprompting the LLM should be handled using other methods. For example jd/tenacity or hynek/stamina to retry a Python function.

"},{"location":"retrying/#retrychatmodel","title":"RetryChatModel","text":"

Under the hood, LLM-assisted retries are implemented using the RetryChatModel which wraps any other ChatModel, catches exceptions, and resubmits them to the LLM. To implement your own retry handling you can follow the pattern of this class. Please file a GitHub issue if you encounter exceptions that should be included in the LLM-assisted retries.

To use the RetryChatModel directly rather than via the max_retries parameter, simply pass it as the model argument to the decorator. Extending the example above

from magentic import OpenaiChatModel\nfrom magentic.chat_model.retry_chat_model import RetryChatModel\n\n\n@prompt(\n    \"Return a country\",\n    model=RetryChatModel(OpenaiChatModel(\"gpt-4o-mini\"), max_retries=3),\n)\ndef get_country() -> Country: ...\n\n\nget_country()\n
"},{"location":"streaming/","title":"Streaming","text":"

The StreamedStr (and AsyncStreamedStr) class can be used to stream the output of the LLM. This allows you to process the text while it is being generated, rather than receiving the whole output at once.

from magentic import prompt, StreamedStr\n\n\n@prompt(\"Tell me about {country}\")\ndef describe_country(country: str) -> StreamedStr: ...\n\n\n# Print the chunks while they are being received\nfor chunk in describe_country(\"Brazil\"):\n    print(chunk, end=\"\")\n# 'Brazil, officially known as the Federative Republic of Brazil, is ...'\n

Multiple StreamedStr can be created at the same time to stream LLM outputs concurrently. In the below example, generating the description for multiple countries takes approximately the same amount of time as for a single country.

from time import time\n\ncountries = [\"Australia\", \"Brazil\", \"Chile\"]\n\n\n# Generate the descriptions one at a time\nstart_time = time()\nfor country in countries:\n    # Converting `StreamedStr` to `str` blocks until the LLM output is fully generated\n    description = str(describe_country(country))\n    print(f\"{time() - start_time:.2f}s : {country} - {len(description)} chars\")\n\n# 22.72s : Australia - 2130 chars\n# 41.63s : Brazil - 1884 chars\n# 74.31s : Chile - 2968 chars\n\n\n# Generate the descriptions concurrently by creating the StreamedStrs at the same time\nstart_time = time()\nstreamed_strs = [describe_country(country) for country in countries]\nfor country, streamed_str in zip(countries, streamed_strs):\n    description = str(streamed_str)\n    print(f\"{time() - start_time:.2f}s : {country} - {len(description)} chars\")\n\n# 22.79s : Australia - 2147 chars\n# 23.64s : Brazil - 2202 chars\n# 24.67s : Chile - 2186 chars\n
"},{"location":"streaming/#object-streaming","title":"Object Streaming","text":"

Structured outputs can also be streamed from the LLM by using the return type annotation Iterable (or AsyncIterable). This allows each item to be processed while the next one is being generated.

from collections.abc import Iterable\nfrom time import time\n\nfrom magentic import prompt\nfrom pydantic import BaseModel\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\n@prompt(\"Create a Superhero team named {name}.\")\ndef create_superhero_team(name: str) -> Iterable[Superhero]: ...\n\n\nstart_time = time()\nfor hero in create_superhero_team(\"The Food Dudes\"):\n    print(f\"{time() - start_time:.2f}s : {hero}\")\n\n# 2.23s : name='Pizza Man' age=30 power='Can shoot pizza slices from his hands' enemies=['The Hungry Horde', 'The Junk Food Gang']\n# 4.03s : name='Captain Carrot' age=35 power='Super strength and agility from eating carrots' enemies=['The Sugar Squad', 'The Greasy Gang']\n# 6.05s : name='Ice Cream Girl' age=25 power='Can create ice cream out of thin air' enemies=['The Hot Sauce Squad', 'The Healthy Eaters']\n
"},{"location":"streaming/#streamedresponse","title":"StreamedResponse","text":"

Some LLMs have the ability to generate text output and make tool calls in the same response. This allows them to perform chain-of-thought reasoning or provide additional context to the user. In magentic, the StreamedResponse (or AsyncStreamedResponse) class can be used to request this type of output. This object is an iterable of StreamedStr (or AsyncStreamedStr) and FunctionCall instances.

Consuming StreamedStr

The StreamedStr object caches its chunks internally, so it does not have to be consumed immediately. This means you can iterate over the chunks as they are received, and/or use the StreamedStr object as a whole after the LLM has finished generating the output.

In the example below, we request that the LLM generates a greeting and then calls a function to get the weather for two cities. The StreamedResponse object is then iterated over to print the output, and the StreamedStr and FunctionCall items are processed separately.

from magentic import prompt, FunctionCall, StreamedResponse, StreamedStr\n\n\ndef get_weather(city: str) -> str:\n    return f\"The weather in {city} is 20\u00b0C.\"\n\n\n@prompt(\n    \"Say hello, then get the weather for: {cities}\",\n    functions=[get_weather],\n)\ndef describe_weather(cities: list[str]) -> StreamedResponse: ...\n\n\nresponse = describe_weather([\"Cape Town\", \"San Francisco\"])\nfor item in response:\n    if isinstance(item, StreamedStr):\n        for chunk in item:\n            # print the chunks as they are received\n            print(chunk, sep=\"\", end=\"\")\n        print()\n    if isinstance(item, FunctionCall):\n        # print the function call, then call it and print the result\n        print(item)\n        print(item())\n\n# Hello! I'll get the weather for Cape Town and San Francisco for you.\n# FunctionCall(<function get_weather at 0x1109825c0>, 'Cape Town')\n# The weather in Cape Town is 20\u00b0C.\n# FunctionCall(<function get_weather at 0x1109825c0>, 'San Francisco')\n# The weather in San Francisco is 20\u00b0C.\n
"},{"location":"structured-outputs/","title":"Structured Outputs","text":""},{"location":"structured-outputs/#pydantic-models","title":"Pydantic Models","text":"

The @prompt decorator will respect the return type annotation of the decorated function. This can be any type supported by pydantic including a pydantic model. See the Pydantic docs for more information about models.

from magentic import prompt\nfrom pydantic import BaseModel\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\n@prompt(\"Create a Superhero named {name}.\")\ndef create_superhero(name: str) -> Superhero: ...\n\n\ncreate_superhero(\"Garden Man\")\n# Superhero(name='Garden Man', age=30, power='Control over plants', enemies=['Pollution Man', 'Concrete Woman'])\n
"},{"location":"structured-outputs/#using-field","title":"Using Field","text":"

With pydantic's BaseModel, you can use Field to provide additional information for individual attributes, such as a description.

from magentic import prompt\nfrom pydantic import BaseModel\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int = Field(\n        description=\"The age of the hero, which could be much older than humans.\"\n    )\n    power: str = Field(examples=[\"Runs really fast\"])\n    enemies: list[str]\n\n\n@prompt(\"Create a Superhero named {name}.\")\ndef create_superhero(name: str) -> Superhero: ...\n
"},{"location":"structured-outputs/#configdict","title":"ConfigDict","text":"

Pydantic also supports configuring the BaseModel by setting the model_config attribute. Magentic extends pydantic's ConfigDict class to add the following additional configuration options

  • openai_strict: bool Indicates whether to use OpenAI's Structured Outputs feature.

See the pydantic Configuration docs for the inherited configuration options.

from magentic import prompt, ConfigDict\nfrom pydantic import BaseModel\n\n\nclass Superhero(BaseModel):\n    model_config = ConfigDict(openai_strict=True)\n\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\n@prompt(\"Create a Superhero named {name}.\")\ndef create_superhero(name: str) -> Superhero: ...\n\n\ncreate_superhero(\"Garden Man\")\n
"},{"location":"structured-outputs/#json-schema","title":"JSON Schema","text":"

OpenAI Structured Outputs

Setting openai_strict=True results in a different JSON schema than that from .model_json_schema() being sent to the LLM. Use openai.pydantic_function_tool(Superhero) to generate the JSON schema in this case.

You can generate the JSON schema for the pydantic model using the .model_json_schema() method. This is what is sent to the LLM.

Running Superhero.model_json_schema() for the above definition reveals the following JSON schema

{\n    \"properties\": {\n        \"name\": {\"title\": \"Name\", \"type\": \"string\"},\n        \"age\": {\n            \"description\": \"The age of the hero, which could be much older than humans.\",\n            \"title\": \"Age\",\n            \"type\": \"integer\",\n        },\n        \"power\": {\n            \"examples\": [\"Runs really fast\"],\n            \"title\": \"Power\",\n            \"type\": \"string\",\n        },\n        \"enemies\": {\"items\": {\"type\": \"string\"}, \"title\": \"Enemies\", \"type\": \"array\"},\n    },\n    \"required\": [\"name\", \"age\", \"power\", \"enemies\"],\n    \"title\": \"Superhero\",\n    \"type\": \"object\",\n}\n

If a StructuredOutputError is raised often, this indicates that the LLM is failing to match the schema. The traceback for these errors includes the underlying pydantic ValidationError which shows in what way the received response was invalid. To combat these errors there are several options

  • Add descriptions or examples for individual fields to demonstrate valid values.
  • Simplify the output schema, including using more flexible types (e.g. str instead of datetime) or allowing fields to be nullable with | None.
  • Switch to a \"more intelligent\" LLM. See Configuration for how to do this.
"},{"location":"structured-outputs/#python-types","title":"Python Types","text":"

Regular Python types can also be used as the function return type.

from magentic import prompt\nfrom pydantic import BaseModel, Field\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\ngarden_man = Superhero(\n    name=\"Garden Man\",\n    age=30,\n    power=\"Control over plants\",\n    enemies=[\"Pollution Man\", \"Concrete Woman\"],\n)\n\n\n@prompt(\"Return True if {hero.name} will be defeated by enemies {hero.enemies}\")\ndef will_be_defeated(hero: Superhero) -> bool: ...\n\n\nhero_defeated = will_be_defeated(garden_man)\nprint(hero_defeated)\n# > True\n
"},{"location":"structured-outputs/#chain-of-thought-prompting","title":"Chain-of-Thought Prompting","text":"

StreamedResponse

It is now recommended to use StreamedResponse for chain-of-thought prompting, as this uses the LLM provider's native chain-of-thought capabilities. See StreamedResponse for more information.

Using a simple Python type as the return annotation might result in poor results as the LLM has no time to arrange its thoughts before answering. To allow the LLM to work through this \"chain of thought\" you can instead return a pydantic model with initial fields for explaining the final response.

from magentic import prompt\nfrom pydantic import BaseModel, Field\n\n\nclass ExplainedDefeated(BaseModel):\n    explanation: str = Field(\n        description=\"Describe the battle between the hero and their enemy.\"\n    )\n    defeated: bool = Field(description=\"True if the hero was defeated.\")\n\n\nclass Superhero(BaseModel):\n    name: str\n    age: int\n    power: str\n    enemies: list[str]\n\n\n@prompt(\"Return True if {hero.name} will be defeated by enemies {hero.enemies}\")\ndef will_be_defeated(hero: Superhero) -> ExplainedDefeated: ...\n\n\ngarden_man = Superhero(\n    name=\"Garden Man\",\n    age=30,\n    power=\"Control over plants\",\n    enemies=[\"Pollution Man\", \"Concrete Woman\"],\n)\n\nhero_defeated = will_be_defeated(garden_man)\nprint(hero_defeated.defeated)\n# > True\nprint(hero_defeated.explanation)\n# > 'Garden Man is an environmental hero who fights against Pollution Man ...'\n
"},{"location":"structured-outputs/#explained","title":"Explained","text":"

Using chain-of-thought is a common approach to improve the output of the model, so a generic Explained model might be generally useful. The description or example parameters of Field can be used to demonstrate the desired style and detail of the explanations.

from typing import Generic, TypeVar\n\nfrom magentic import prompt\nfrom pydantic import BaseModel, Field\n\n\nT = TypeVar(\"T\")\n\n\nclass Explained(BaseModel, Generic[T]):\n    explanation: str = Field(description=\"Explanation of how the value was determined.\")\n    value: T\n\n\n@prompt(\"Return True if {hero.name} will be defeated by enemies {hero.enemies}\")\ndef will_be_defeated(hero: Superhero) -> Explained[bool]: ...\n
"},{"location":"type-checking/","title":"Type Checking","text":"

Many type checkers will raise warnings or errors for functions with the @prompt decorator due to the function having no body or return value. There are several ways to deal with these.

  1. Disable the check globally for the type checker. For example in mypy by disabling error code empty-body. 
    # pyproject.toml\n[tool.mypy]\ndisable_error_code = [\"empty-body\"]\n
  2. Make the function body ... (this does not satisfy mypy) or raise. 
    @prompt(\"Choose a color\")\ndef random_color() -> str: ...\n
  3. Use comment # type: ignore[empty-body] on each function. In this case you can add a docstring instead of .... 
    @prompt(\"Choose a color\")\ndef random_color() -> str:  # type: ignore[empty-body]\n    \"\"\"Returns a random color.\"\"\"\n
"},{"location":"vision/","title":"Vision","text":"

Image inputs can be provided to LLMs in magentic by using the UserImageMessage message type.

Anthropic Image URLs

Anthropic models currently do not support supplying an image as a url, just bytes.

For more information visit the OpenAI Vision API documentation or the Anthropic Vision API documentation.

"},{"location":"vision/#userimagemessage","title":"UserImageMessage","text":"

The UserImageMessage can be used in @chatprompt alongside other messages. The LLM must be set to an OpenAI or Anthropic model that supports vision, for example gpt-4o (the default ChatModel). This can be done by passing the model parameter to @chatprompt, or through the other methods of configuration.

from pydantic import BaseModel, Field\n\nfrom magentic import chatprompt, UserMessage\nfrom magentic.vision import UserImageMessage\n\n\nIMAGE_URL_WOODEN_BOARDWALK = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\"\n\n\nclass ImageDetails(BaseModel):\n    description: str = Field(description=\"A brief description of the image.\")\n    name: str = Field(description=\"A short name.\")\n\n\n@chatprompt(\n    UserMessage(\"Describe the following image in one sentence.\"),\n    UserImageMessage(IMAGE_URL_WOODEN_BOARDWALK),\n)\ndef describe_image() -> ImageDetails: ...\n\n\nimage_details = describe_image()\nprint(image_details.name)\n# 'Wooden Boardwalk in Green Wetland'\nprint(image_details.description)\n# 'A serene wooden boardwalk meanders through a lush green wetland under a blue sky dotted with clouds.'\n

For more info on the @chatprompt decorator, see Chat Prompting.

"},{"location":"vision/#placeholder","title":"Placeholder","text":"

In the previous example, the image url was tied to the function. To provide the image as a function parameter, use Placeholder. This substitutes a function argument into the message when the function is called.

from magentic import chatprompt, Placeholder, UserMessage\nfrom magentic.vision import UserImageMessage\n\n\nIMAGE_URL_WOODEN_BOARDWALK = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\"\n\n\n@chatprompt(\n    UserMessage(\"Describe the following image in one sentence.\"),\n    UserImageMessage(Placeholder(str, \"image_url\")),\n)\ndef describe_image(image_url: str) -> str: ...\n\n\ndescribe_image(IMAGE_URL_WOODEN_BOARDWALK)\n# 'A wooden boardwalk meanders through lush green wetlands under a partly cloudy blue sky.'\n
"},{"location":"vision/#bytes","title":"bytes","text":"

UserImageMessage can also accept bytes as input. Like str, this can be passed directly or via Placeholder.

import requests\n\nfrom magentic import chatprompt, Placeholder, UserMessage\nfrom magentic.vision import UserImageMessage\n\n\nIMAGE_URL_WOODEN_BOARDWALK = \"https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg\"\n\n\ndef url_to_bytes(url: str) -> bytes:\n    \"\"\"Get the content of a URL as bytes.\"\"\"\n\n    # A custom user-agent is necessary to comply with Wikimedia user-agent policy\n    # https://meta.wikimedia.org/wiki/User-Agent_policy\n    headers = {\"User-Agent\": \"MagenticExampleBot (https://magentic.dev/)\"}\n    return requests.get(url, headers=headers, timeout=10).content\n\n\n@chatprompt(\n    UserMessage(\"Describe the following image in one sentence.\"),\n    UserImageMessage(Placeholder(bytes, \"image_bytes\")),\n)\ndef describe_image(image_bytes: bytes) -> str: ...\n\n\nimage_bytes = url_to_bytes(IMAGE_URL_WOODEN_BOARDWALK)\ndescribe_image(image_bytes)\n# 'The image shows a wooden boardwalk extending through a lush green wetland with a backdrop of blue skies and scattered clouds.'\n
"},{"location":"examples/chain_of_verification/","title":"Chain of Verification (CoVe)","text":"In\u00a0[1]: Copied! 
# Define the prompts\n\nimport asyncio\n\nfrom magentic import prompt\n\n\n@prompt(\"{query}\")\nasync def answer_query(query: str) -> str: ...\n\n\n@prompt(\n    \"\"\"\\\nQuery: {query}\nResponse: {response}\n\nProvide specific questions to verify the facts in the above response as related to the query.\n\"\"\"\n)\nasync def generate_verification_questions(query: str, response: str) -> list[str]: ...\n\n\n@prompt(\n    \"\"\"\\\n{context}\n\nGiven the above context, what is the answer to the following question?\n\n{query}\"\"\"\n)\nasync def answer_query_with_context(query: str, context: str) -> str: ...\n
# Define the prompts import asyncio from magentic import prompt @prompt(\"{query}\") async def answer_query(query: str) -> str: ... @prompt( \"\"\"\\ Query: {query} Response: {response} Provide specific questions to verify the facts in the above response as related to the query. \"\"\" ) async def generate_verification_questions(query: str, response: str) -> list[str]: ... @prompt( \"\"\"\\ {context} Given the above context, what is the answer to the following question? {query}\"\"\" ) async def answer_query_with_context(query: str, context: str) -> str: ... In\u00a0[2]: Copied! 
# 1. Generate Baseline Response\n# Given a query, generate the response using the LLM.\n\nquery = \"Name some politicians who were born in NY, New York\"\n\nbaseline_response = await answer_query(query)\nprint(baseline_response)\n
# 1. Generate Baseline Response # Given a query, generate the response using the LLM. query = \"Name some politicians who were born in NY, New York\" baseline_response = await answer_query(query) print(baseline_response) 
Here are some politicians who were born in New York, New York:\n\n1. Franklin D. Roosevelt - 32nd President of the United States.\n2. Theodore Roosevelt - 26th President of the United States.\n3. Donald Trump - 45th President of the United States.\n4. Hillary Clinton - Former Secretary of State and Democratic nominee for President in 2016.\n5. Michael Bloomberg - Former Mayor of New York City and businessman.\n6. Rudy Giuliani - Former Mayor of New York City and attorney.\n7. Chuck Schumer - U.S. Senator from New York and current Senate Majority Leader.\n8. Kirsten Gillibrand - U.S. Senator from New York.\n9. Mario Cuomo - Former Governor of New York.\n10. Andrew Cuomo - Current Governor of New York.\n\nPlease note that this is not an exhaustive list, and there are many more politicians who were born in New York, New York.\n
In\u00a0[3]: Copied! 
# 2. Plan Verifications\n# Given both query and baseline response, generate a list of verification questions\n# that could help to self-analyze if there are any mistakes in the original response.\n\nverification_questions = await generate_verification_questions(query, baseline_response)\n\nfor q in verification_questions:\n    print(q)\n
# 2. Plan Verifications # Given both query and baseline response, generate a list of verification questions # that could help to self-analyze if there are any mistakes in the original response. verification_questions = await generate_verification_questions(query, baseline_response) for q in verification_questions: print(q) 
Was Franklin D. Roosevelt born in New York, New York?\nWas Theodore Roosevelt born in New York, New York?\nWas Donald Trump born in New York, New York?\nWas Hillary Clinton born in New York, New York?\nWas Michael Bloomberg born in New York, New York?\nWas Rudy Giuliani born in New York, New York?\nWas Chuck Schumer born in New York, New York?\nWas Kirsten Gillibrand born in New York, New York?\nWas Mario Cuomo born in New York, New York?\nIs Andrew Cuomo the current Governor of New York?\n
In\u00a0[4]: Copied! 
# 3. Execute Verifications\n# Answer each verification question in turn, and hence check the answer against the\n# original response to check for inconsistencies or mistakes.\n\nverification_answers = await asyncio.gather(\n    *(answer_query(question) for question in verification_questions)\n)\n\nfor ans in verification_answers:\n    print(ans)\n
# 3. Execute Verifications # Answer each verification question in turn, and hence check the answer against the # original response to check for inconsistencies or mistakes. verification_answers = await asyncio.gather( *(answer_query(question) for question in verification_questions) ) for ans in verification_answers: print(ans) 
Yes, Franklin D. Roosevelt was born on January 30, 1882, in Hyde Park, New York, which is located in Dutchess County.\nYes, Theodore Roosevelt was born in New York City, New York on October 27, 1858. Specifically, he was born in a house located at 28 East 20th Street in Manhattan.\nYes, Donald Trump was indeed born in New York, New York on June 14, 1946.\nNo, Hillary Clinton was not born in New York, New York. She was born on October 26, 1947, in Chicago, Illinois.\nNo, Michael Bloomberg was born in Boston, Massachusetts on February 14, 1942.\nYes, Rudy Giuliani was born in New York, New York. He was born on May 28, 1944, in the Brooklyn borough of New York City.\nYes, Chuck Schumer was born in Brooklyn, New York on November 23, 1950.\nNo, Kirsten Gillibrand was born in Albany, New York on December 9, 1966.\nYes, Mario Cuomo was born in New York City, New York, United States. He was born on June 15, 1932, in the borough of Queens, specifically in the neighborhood of South Jamaica.\nAs of September 2021, Andrew Cuomo is the current Governor of New York. However, please note that political positions can change, and it is always recommended to verify the information with up-to-date sources.\n
In\u00a0[5]: Copied! 
# 4. Generate Final Verified Response\n# Given the discovered inconsistencies (if any), generate a revised response\n# incorporating the verification results.\n\ncontext = \"\\n\".join(verification_answers)\nverified_response = await answer_query_with_context(query, context)\nprint(verified_response)\n
# 4. Generate Final Verified Response # Given the discovered inconsistencies (if any), generate a revised response # incorporating the verification results. context = \"\\n\".join(verification_answers) verified_response = await answer_query_with_context(query, context) print(verified_response) 
Some politicians who were born in New York, New York include Franklin D. Roosevelt, Theodore Roosevelt, Donald Trump, Rudy Giuliani, and Mario Cuomo.\n
"},{"location":"examples/chain_of_verification/#chain-of-verification-cove","title":"Chain of Verification (CoVe)\u00b6","text":"

This notebook is a basic implementation of the paper Chain-of-Verification Reduces Hallucination In Large Language Models (2023) (arXiv: [2309.11495]).

"},{"location":"examples/rag_github/","title":"Retrieval-Augmented Generation with GitHub","text":"In\u00a0[\u00a0]: Copied! 
# Install dependencies (skip this cell if already installed)\n! pip install magentic\n! pip install ghapi\n
# Install dependencies (skip this cell if already installed) ! pip install magentic ! pip install ghapi In\u00a0[2]: Copied! 
# Configure magentic to use the `gpt-3.5-turbo` model for this notebook\n%env MAGENTIC_OPENAI_MODEL=gpt-3.5-turbo\n
# Configure magentic to use the `gpt-3.5-turbo` model for this notebook %env MAGENTIC_OPENAI_MODEL=gpt-3.5-turbo 
env: MAGENTIC_OPENAI_MODEL=gpt-3.5-turbo\n

Let's start by creating a prompt-function to generate some text recommending GitHub repos for a topic.

In\u00a0[3]: Copied! 
# Create a prompt-function to describe the latest GitHub repos\n\nfrom IPython.display import Markdown, display\n\nfrom magentic import prompt\n\n\n@prompt(\n    \"\"\"What are the latest github repos I should use related to {topic}?\n    Recommend three in particular that I should check out and why.\n    Provide a link to each, and a note on whether they are actively maintained.\n    \"\"\"\n)\ndef recommmend_github_repos(topic: str) -> str: ...\n\n\noutput = recommmend_github_repos(\"LLMs\")\ndisplay(Markdown(output))\n
# Create a prompt-function to describe the latest GitHub repos from IPython.display import Markdown, display from magentic import prompt @prompt( \"\"\"What are the latest github repos I should use related to {topic}? Recommend three in particular that I should check out and why. Provide a link to each, and a note on whether they are actively maintained. \"\"\" ) def recommmend_github_repos(topic: str) -> str: ... output = recommmend_github_repos(\"LLMs\") display(Markdown(output))
  1. Hugging Face Transformers: This repository contains a library for Natural Language Processing (NLP) tasks using the latest Transformer models, including LLMs. It is actively maintained by Hugging Face, a popular NLP research group, and has a large community contributing to it.

Link: https://github.com/huggingface/transformers

  1. OpenAI GPT-3: This repository contains the code for OpenAI's GPT-3 model, one of the most advanced LLMs available. While the repository may not be frequently updated due to proprietary restrictions, it provides valuable insights into how state-of-the-art LLMs are implemented.

Link: https://github.com/openai/gpt-3

  1. AllenNLP: AllenNLP is a deep learning library for NLP research that provides easy-to-use tools for building and experimenting with LLMs. The repository is actively maintained by the Allen Institute for AI and offers a wide range of pre-trained models, including BERT and GPT-2.

Link: https://github.com/allenai/allennlp

Please note that the availability and maintenance status of these repositories may change over time, so it's a good idea to check for the latest updates before diving in.

The LLM has no knowledge of GitHub repos created after its knowledge cutoff date! Also, it occasionally hallucinates some of its answers. To solve these issues we need to provide it with up-to-date information in the prompt, which it can use to generate an informed answer.

First we'll create a function for searching for GitHub repos.

In\u00a0[4]: Copied! 
# Create a function to search for GitHub repos\n\nfrom ghapi.all import GhApi\nfrom pydantic import BaseModel\n\ngithub = GhApi(authenticate=False)\n\n\nclass GithubRepo(BaseModel):\n    full_name: str\n    description: str\n    html_url: str\n    stargazers_count: int\n    pushed_at: str\n\n\ndef search_github_repos(query: str, num_results: int = 10):\n    results = github.search.repos(query, per_page=num_results)\n    return [GithubRepo.model_validate(item) for item in results[\"items\"]]\n
# Create a function to search for GitHub repos from ghapi.all import GhApi from pydantic import BaseModel github = GhApi(authenticate=False) class GithubRepo(BaseModel): full_name: str description: str html_url: str stargazers_count: int pushed_at: str def search_github_repos(query: str, num_results: int = 10): results = github.search.repos(query, per_page=num_results) return [GithubRepo.model_validate(item) for item in results[\"items\"]] In\u00a0[5]: Copied! 
# Test that github search works\n\nfor item in search_github_repos(\"openai\", num_results=3):\n    print(item.model_dump_json(indent=2))\n
# Test that github search works for item in search_github_repos(\"openai\", num_results=3): print(item.model_dump_json(indent=2)) 
{\n  \"full_name\": \"openai/openai-cookbook\",\n  \"description\": \"Examples and guides for using the OpenAI API\",\n  \"html_url\": \"https://github.com/openai/openai-cookbook\",\n  \"stargazers_count\": 55805,\n  \"pushed_at\": \"2024-04-19T19:05:02Z\"\n}\n{\n  \"full_name\": \"betalgo/openai\",\n  \"description\": \"OpenAI .NET sdk - Azure OpenAI, ChatGPT, Whisper,  and DALL-E \",\n  \"html_url\": \"https://github.com/betalgo/openai\",\n  \"stargazers_count\": 2721,\n  \"pushed_at\": \"2024-04-20T22:50:28Z\"\n}\n{\n  \"full_name\": \"openai/openai-python\",\n  \"description\": \"The official Python library for the OpenAI API\",\n  \"html_url\": \"https://github.com/openai/openai-python\",\n  \"stargazers_count\": 19786,\n  \"pushed_at\": \"2024-04-21T01:04:42Z\"\n}\n

Now, we can provide the results of the search as context to the LLM to create an improved recommmend_github_repos function.

In\u00a0[6]: Copied! 
# Combine the search with a prompt-function to describe the latest GitHub repos\n\nfrom magentic import prompt\n\n\n@prompt(\n    \"\"\"What are the latest github repos I should use related to {topic}?\n    Recommend three in particular that I should check out and why.\n    Provide a link to each, and a note on whether they are actively maintained.\n\n    Here are the latest search results for this topic on GitHub:\n    {search_results}\n    \"\"\",\n)\ndef recommmend_github_repos_using_search_results(\n    topic: str, search_results: list[GithubRepo]\n) -> str: ...\n\n\ndef recommmend_github_repos(topic: str) -> str:\n    search_results = search_github_repos(topic, num_results=10)\n    return recommmend_github_repos_using_search_results(topic, search_results)\n\n\noutput = recommmend_github_repos(\"LLMs\")\ndisplay(Markdown(output))\n
# Combine the search with a prompt-function to describe the latest GitHub repos from magentic import prompt @prompt( \"\"\"What are the latest github repos I should use related to {topic}? Recommend three in particular that I should check out and why. Provide a link to each, and a note on whether they are actively maintained. Here are the latest search results for this topic on GitHub: {search_results} \"\"\", ) def recommmend_github_repos_using_search_results( topic: str, search_results: list[GithubRepo] ) -> str: ... def recommmend_github_repos(topic: str) -> str: search_results = search_github_repos(topic, num_results=10) return recommmend_github_repos_using_search_results(topic, search_results) output = recommmend_github_repos(\"LLMs\") display(Markdown(output))

Based on the latest search results, here are three GitHub repos related to Large Language Models (LLMs) that you should check out:

  1. gpt4all:

    • Description: gpt4all: run open-source LLMs anywhere
    • Stargazers Count: 63,790
    • Last Pushed: 2024-04-19
    • Active Maintenance: Yes

  2. LLaMA-Factory:

    • Description: Unify Efficient Fine-Tuning of 100+ LLMs
    • Stargazers Count: 17,047
    • Last Pushed: 2024-04-21
    • Active Maintenance: Yes

  3. LLMsPracticalGuide:

    • Description: A curated list of practical guide resources of LLMs (LLMs Tree, Examples, Papers)
    • Stargazers Count: 8,484
    • Last Pushed: 2024-01-10
    • Active Maintenance: It seems less actively maintained compared to the other two repos, but still worth checking out.

These repos cover a range of topics related to LLMs and can provide valuable resources and tools for your projects.

Now the answer contains up-to-date and correct information!

"},{"location":"examples/rag_github/#retrieval-augmented-generation-with-github","title":"Retrieval-Augmented Generation with GitHub\u00b6","text":"

This notebook demonstrates how to perform Retrieval-Augmented Generation (RAG) with magentic using the GitHub API. Essentially, RAG provides context to the LLM which it can use when generating its response. This approach allows us to insert new or private information that was not present in the model's training data.

"},{"location":"examples/registering_custom_type/","title":"Registering a Custom Type","text":"In\u00a0[1]: Copied! 
# Create FunctionSchema for pd.DataFrame\n\nimport json\nfrom collections.abc import Iterable\nfrom typing import Any\n\nimport pandas as pd\n\nfrom magentic.chat_model.function_schema import FunctionSchema, register_function_schema\n\n\n@register_function_schema(pd.DataFrame)\nclass DataFrameFunctionSchema(FunctionSchema[pd.DataFrame]):\n    @property\n    def name(self) -> str:\n        \"\"\"The name of the function.\n\n        Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64.\n        \"\"\"\n        return \"dataframe\"\n\n    @property\n    def description(self) -> str | None:\n        return \"A DataFrame object.\"\n\n    @property\n    def parameters(self) -> dict[str, Any]:\n        \"\"\"The parameters the functions accepts as a JSON Schema object.\"\"\"\n        return {\n            \"properties\": {\n                \"columns\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}},\n                \"data\": {\n                    \"type\": \"array\",\n                    \"items\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}},\n                },\n            },\n            \"required\": [\"index\", \"columns\", \"data\"],\n            \"type\": \"object\",\n        }\n\n    def parse_args(self, chunks: Iterable[str]) -> pd.DataFrame:\n        \"\"\"Parse an iterable of string chunks into the function arguments.\"\"\"\n        args = json.loads(\"\".join(chunks))\n        return pd.DataFrame(**args)\n\n    def serialize_args(self, value: pd.DataFrame) -> dict:\n        \"\"\"Serialize an object into a JSON string of function arguments.\"\"\"\n        return {\n            \"columns\": value.columns.tolist(),\n            \"data\": value.to_numpy().tolist(),\n        }\n
# Create FunctionSchema for pd.DataFrame import json from collections.abc import Iterable from typing import Any import pandas as pd from magentic.chat_model.function_schema import FunctionSchema, register_function_schema @register_function_schema(pd.DataFrame) class DataFrameFunctionSchema(FunctionSchema[pd.DataFrame]): @property def name(self) -> str: \"\"\"The name of the function. Must be a-z, A-Z, 0-9, or contain underscores and dashes, with a maximum length of 64. \"\"\" return \"dataframe\" @property def description(self) -> str | None: return \"A DataFrame object.\" @property def parameters(self) -> dict[str, Any]: \"\"\"The parameters the functions accepts as a JSON Schema object.\"\"\" return { \"properties\": { \"columns\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}}, \"data\": { \"type\": \"array\", \"items\": {\"type\": \"array\", \"items\": {\"type\": \"string\"}}, }, }, \"required\": [\"index\", \"columns\", \"data\"], \"type\": \"object\", } def parse_args(self, chunks: Iterable[str]) -> pd.DataFrame: \"\"\"Parse an iterable of string chunks into the function arguments.\"\"\" args = json.loads(\"\".join(chunks)) return pd.DataFrame(**args) def serialize_args(self, value: pd.DataFrame) -> dict: \"\"\"Serialize an object into a JSON string of function arguments.\"\"\" return { \"columns\": value.columns.tolist(), \"data\": value.to_numpy().tolist(), } In\u00a0[2]: Copied! 
# Roundtrip test the new FunctionSchema\n\nfunction_schema = DataFrameFunctionSchema(pd.DataFrame)\n\ndf_test = pd.DataFrame(\n    {\n        \"A\": [1, 2, 3],\n        \"B\": [4, 5, 6],\n    },\n)\n\nargs = function_schema.serialize_args(df_test)\nprint(args)\n\nobj = function_schema.parse_args(json.dumps(args))\nobj\n
# Roundtrip test the new FunctionSchema function_schema = DataFrameFunctionSchema(pd.DataFrame) df_test = pd.DataFrame( { \"A\": [1, 2, 3], \"B\": [4, 5, 6], }, ) args = function_schema.serialize_args(df_test) print(args) obj = function_schema.parse_args(json.dumps(args)) obj 
{'columns': ['A', 'B'], 'data': [[1, 4], [2, 5], [3, 6]]}\n
Out[2]: A B 0 1 4 1 2 5 2 3 6 In\u00a0[3]: Copied! 
# Use pd.DataFrame as the return type of a prompt function\n\nimport pandas as pd\n\nfrom magentic import prompt\n\n\n@prompt(\n    \"Create a table listing the ingredients needed to cook {dish}.\"\n    \"Include a column for the quantity of each ingredient.\"\n    \"Also include a column with alergy information.\"\n)\ndef list_ingredients(dish: str) -> pd.DataFrame: ...\n\n\nlist_ingredients(\"lasagna\")\n
# Use pd.DataFrame as the return type of a prompt function import pandas as pd from magentic import prompt @prompt( \"Create a table listing the ingredients needed to cook {dish}.\" \"Include a column for the quantity of each ingredient.\" \"Also include a column with alergy information.\" ) def list_ingredients(dish: str) -> pd.DataFrame: ... list_ingredients(\"lasagna\") Out[3]: Ingredient Quantity Allergy Information 0 Lasagna noodles 16 oz Contains wheat, may contain egg and soy 1 Ground beef 1 lb Contains beef, may contain soy and gluten 2 Tomato sauce 24 oz Contains tomatoes, may contain soy and garlic 3 Mozzarella cheese 16 oz Contains milk, may contain soy 4 Ricotta cheese 15 oz Contains milk, may contain soy and eggs 5 Parmesan cheese 1 cup Contains milk, may contain soy and eggs 6 Garlic 3 cloves No known allergies 7 Onion 1 No known allergies 8 Olive oil 2 tbsp No known allergies 9 Salt 1 tsp No known allergies 10 Pepper 1/2 tsp No known allergies 11 Italian seasoning 1 tsp No known allergies 12 Sugar 1 tsp No known allergies"},{"location":"examples/registering_custom_type/#registering-a-custom-type","title":"Registering a Custom Type\u00b6","text":"

This notebook shows how to register a new type so that it can be used as the return annotation for @prompt, @promptchain, and @chatprompt. This is done by creating a new FunctionSchema which defines the parameters required to create the type, and how to parse/serialize these from/to the LLM.

See https://platform.openai.com/docs/guides/function-calling for more information on function calling, which enables this.

"},{"location":"examples/vision_renaming_screenshots/","title":"Renaming Screenshots with GPT Vision","text":"In\u00a0[1]: Copied! 
# List all screenshots\n\nfrom pathlib import Path\n\npath_desktop = Path.home() / \"Desktop\"\nscreenshot_paths = list(path_desktop.glob(\"Screenshot*.png\"))\n\nfor screenshot_path in screenshot_paths:\n    print(screenshot_path.name)\n
# List all screenshots from pathlib import Path path_desktop = Path.home() / \"Desktop\" screenshot_paths = list(path_desktop.glob(\"Screenshot*.png\")) for screenshot_path in screenshot_paths: print(screenshot_path.name) 
Screenshot 2024-04-20 at 10.49.08\u202fPM Small.png\nScreenshot 2024-04-20 at 10.50.04\u202fPM Small.png\nScreenshot 2024-04-20 at 10.50.57\u202fPM Small.png\n
In\u00a0[2]: Copied! 
# Display the first screenshot\n\nfrom IPython.display import Image, display\n\n\ndef diplay_image(image_path):\n    display(Image(data=image_path.read_bytes(), width=400))\n\n\ndiplay_image(screenshot_paths[0])\n
# Display the first screenshot from IPython.display import Image, display def diplay_image(image_path): display(Image(data=image_path.read_bytes(), width=400)) diplay_image(screenshot_paths[0]) In\u00a0[3]: Copied! 
# Define desired output for each screenshot\n# Include a description field to allow the LLM to think before naming the file\n\nfrom pydantic import BaseModel, Field\n\n\nclass ScreenshotDetails(BaseModel):\n    description: str = Field(\n        description=\"A brief description of the screenshot, including details that will be useful for naming it.\"\n    )\n    filename: str = Field(\n        description=\"An appropriate file name for this image, excluding the file extension.\"\n    )\n
# Define desired output for each screenshot # Include a description field to allow the LLM to think before naming the file from pydantic import BaseModel, Field class ScreenshotDetails(BaseModel): description: str = Field( description=\"A brief description of the screenshot, including details that will be useful for naming it.\" ) filename: str = Field( description=\"An appropriate file name for this image, excluding the file extension.\" ) In\u00a0[4]: Copied! 
# Create a prompt-function to return details given an image\n\nfrom magentic import OpenaiChatModel, Placeholder, UserMessage, chatprompt\nfrom magentic.vision import UserImageMessage\n\n\n@chatprompt(\n    UserMessage(\"Describe the screenshot, then provide a suitable file name.\"),\n    UserImageMessage(Placeholder(bytes, \"image\")),\n    model=OpenaiChatModel(\"gpt-4-turbo\"),\n)\ndef describe_image(image: bytes) -> ScreenshotDetails: ...\n
# Create a prompt-function to return details given an image from magentic import OpenaiChatModel, Placeholder, UserMessage, chatprompt from magentic.vision import UserImageMessage @chatprompt( UserMessage(\"Describe the screenshot, then provide a suitable file name.\"), UserImageMessage(Placeholder(bytes, \"image\")), model=OpenaiChatModel(\"gpt-4-turbo\"), ) def describe_image(image: bytes) -> ScreenshotDetails: ... In\u00a0[5]: Copied! 
# Rename all screenshots using the prompt-function\n\nfor path_screenshot in path_desktop.glob(\"Screenshot*.png\"):\n    print(path_screenshot.name)\n    diplay_image(path_screenshot)\n\n    image_bytes = path_screenshot.read_bytes()\n    image_details = describe_image(image_bytes)\n    print(image_details.description)\n\n    new_path = path_screenshot.with_stem(image_details.filename)\n    path_screenshot.rename(new_path)\n    print(\"\\nRenamed to:\", new_path.name)\n    print(\"\\n\\n---\\n\\n\")\n
# Rename all screenshots using the prompt-function for path_screenshot in path_desktop.glob(\"Screenshot*.png\"): print(path_screenshot.name) diplay_image(path_screenshot) image_bytes = path_screenshot.read_bytes() image_details = describe_image(image_bytes) print(image_details.description) new_path = path_screenshot.with_stem(image_details.filename) path_screenshot.rename(new_path) print(\"\\nRenamed to:\", new_path.name) print(\"\\n\\n---\\n\\n\") 
Screenshot 2024-04-20 at 10.49.08\u202fPM Small.png\n
The image shows the face of a white alpaca looking directly at the camera. The alpaca has a unique and stylish mohawk-like hairstyle. The background is blurred with a hint of green, suggesting an outdoor setting, likely a field.\n\nRenamed to: stylish-alpaca-face.png\n\n\n---\n\n\nScreenshot 2024-04-20 at 10.50.04\u202fPM Small.png\n
A close-up image of a vibrant green snake coiled around a tree branch. The snake features a beautiful pattern of yellow spots and has a focused gaze. The background is softly blurred, emphasizing the snake in the foreground.\n\nRenamed to: green_snake_coiled_on_branch.png\n\n\n---\n\n\nScreenshot 2024-04-20 at 10.50.57\u202fPM Small.png\n
The image displays a close-up view of a serving of lasagna on a white plate. The lasagna appears richly layered with melted cheese on top and a golden-brown crust, suggesting it is freshly baked and possibly contains a meaty sauce between the pasta sheets.\n\nRenamed to: close_up_lasagna_on_plate.png\n\n\n---\n\n\n
In\u00a0[\u00a0]: Copied! 
\n
"},{"location":"examples/vision_renaming_screenshots/#renaming-screenshots-with-gpt-vision","title":"Renaming Screenshots with GPT Vision\u00b6","text":"

This notebook demonstrates how to use the vision capabilites of gpt-4-turbo in combination with magentic's structured outputs to rename all those screenshots cluttering your desktop.

"}]} \ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml index 2da6878d..f3375086 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -2,66 +2,66 @@ https://magentic.dev/ - 2024-11-30 + 2024-12-01 https://magentic.dev/asyncio/ - 2024-11-30 + 2024-12-01 https://magentic.dev/chat-prompting/ - 2024-11-30 + 2024-12-01 https://magentic.dev/configuration/ - 2024-11-30 + 2024-12-01 https://magentic.dev/formatting/ - 2024-11-30 + 2024-12-01 https://magentic.dev/function-calling/ - 2024-11-30 + 2024-12-01 https://magentic.dev/logging-and-tracing/ - 2024-11-30 + 2024-12-01 https://magentic.dev/retrying/ - 2024-11-30 + 2024-12-01 https://magentic.dev/streaming/ - 2024-11-30 + 2024-12-01 https://magentic.dev/structured-outputs/ - 2024-11-30 + 2024-12-01 https://magentic.dev/type-checking/ - 2024-11-30 + 2024-12-01 https://magentic.dev/vision/ - 2024-11-30 + 2024-12-01 https://magentic.dev/examples/chain_of_verification/ - 2024-11-30 + 2024-12-01 https://magentic.dev/examples/rag_github/ - 2024-11-30 + 2024-12-01 https://magentic.dev/examples/registering_custom_type/ - 2024-11-30 + 2024-12-01 https://magentic.dev/examples/vision_renaming_screenshots/ - 2024-11-30 + 2024-12-01 \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index a2c6d9c9..67fd5e91 100644 Binary files a/sitemap.xml.gz and b/sitemap.xml.gz differ diff --git a/streaming/index.html b/streaming/index.html index 8139abb2..dfe1d33b 100644 --- a/streaming/index.html +++ b/streaming/index.html @@ -839,9 +839,9 @@

Object StreamingStreamedResponse

Some LLMs have the ability to generate text output and make tool calls in the same response. This allows them to perform chain-of-thought reasoning or provide additional context to the user. In magentic, the StreamedResponse (or AsyncStreamedResponse) class can be used to request this type of output. This object is an iterable of StreamedStr (or AsyncStreamedStr) and FunctionCall instances.

-
+

Consuming StreamedStr

-

The StreamedStr object must be iterated over before the next item in the StreamedResponse is processed, otherwise the string output will be lost. This is because the StreamedResponse and StreamedStr share the same underlying generator, so advancing the StreamedResponse iterator skips over the StreamedStr items. The StreamedStr object has internal caching so after iterating over it once the chunks will remain available.

+

The StreamedStr object caches its chunks internally, so it does not have to be consumed immediately. This means you can iterate over the chunks as they are received, and/or use the StreamedStr object as a whole after the LLM has finished generating the output.

In the example below, we request that the LLM generates a greeting and then calls a function to get the weather for two cities. The StreamedResponse object is then iterated over to print the output, and the StreamedStr and FunctionCall items are processed separately.

from magentic import prompt, FunctionCall, StreamedResponse, StreamedStr
@@ -952,6 +952,32 @@ 
StreamedResponse
+  
+    
+    
+    
+    
+      
+      
+    
+    
+      
+    
+  
+    
+    
+    
+    
+      
+      
+    
+    
+      
+    
+
diff --git a/structured-outputs/index.html b/structured-outputs/index.html index 08c277e4..6beb543a 100644 --- a/structured-outputs/index.html +++ b/structured-outputs/index.html @@ -1009,7 +1009,7 @@

Python TypesChain-of-Thought Prompting

StreamedResponse

-

It is now recommended to use StreamedResponse for chain-of-thought prompting, as this uses the LLM provider's native chain-of-thought capabilities. See StreamedResponse for more information.

+

It is now recommended to use StreamedResponse for chain-of-thought prompting, as this uses the LLM provider's native chain-of-thought capabilities. See StreamedResponse for more information.

Using a simple Python type as the return annotation might result in poor results as the LLM has no time to arrange its thoughts before answering. To allow the LLM to work through this "chain of thought" you can instead return a pydantic model with initial fields for explaining the final response.

from magentic import prompt
@@ -1142,6 +1142,32 @@ 
Explained
+  
+    
+    
+    
+    
+      
+      
+    
+    
+      
+    
+  
+    
+    
+    
+    
+      
+      
+    
+    
+      
+    
+
diff --git a/type-checking/index.html b/type-checking/index.html index a0a9c721..b90016a4 100644 --- a/type-checking/index.html +++ b/type-checking/index.html @@ -784,6 +784,32 @@

Type Checking + + +
+ + + + + + + + + + + + + + + + + + + + + + +
diff --git a/vision/index.html b/vision/index.html index 7d05012e..84dcb780 100644 --- a/vision/index.html +++ b/vision/index.html @@ -947,6 +947,32 @@

bytes& Material for MkDocs + + +
+ + + + + + + + + + + + + + + + + + + + + + +