deploy: 0b4b795

jackmpcollins · Nov 30, 2024 · 1c0c38d · 1c0c38d
1 parent 509f8a1
commit 1c0c38d
Show file tree

Hide file tree

Showing 5 changed files with 78 additions and 17 deletions.
diff --git a/search/search_index.json b/search/search_index.json
diff --git a/sitemap.xml b/sitemap.xml
@@ -2,66 +2,66 @@
 <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
     <url>
          <loc>https://magentic.dev/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/asyncio/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/chat-prompting/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/configuration/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/formatting/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/function-calling/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/logging-and-tracing/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/retrying/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/streaming/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/structured-outputs/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/type-checking/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/vision/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/examples/chain_of_verification/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/examples/rag_github/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/examples/registering_custom_type/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
     <url>
          <loc>https://magentic.dev/examples/vision_renaming_screenshots/</loc>
-         <lastmod>2024-11-29</lastmod>
+         <lastmod>2024-11-30</lastmod>
     </url>
 </urlset>
diff --git a/sitemap.xml.gz b/sitemap.xml.gz
diff --git a/streaming/index.html b/streaming/index.html
@@ -450,6 +450,15 @@
     </span>
   </a>
 
+</li>
+
+        <li class="md-nav__item">
+  <a href="#streamedresponse" class="md-nav__link">
+    <span class="md-ellipsis">
+      StreamedResponse
+    </span>
+  </a>
+
 </li>
 
     </ul>
@@ -723,6 +732,15 @@
     </span>
   </a>
 
+</li>
+
+        <li class="md-nav__item">
+  <a href="#streamedresponse" class="md-nav__link">
+    <span class="md-ellipsis">
+      StreamedResponse
+    </span>
+  </a>
+
 </li>
 
     </ul>
@@ -819,6 +837,45 @@ <h2 id="object-streaming">Object Streaming<a class="headerlink" href="#object-st
 <span class="c1"># 4.03s : name=&#39;Captain Carrot&#39; age=35 power=&#39;Super strength and agility from eating carrots&#39; enemies=[&#39;The Sugar Squad&#39;, &#39;The Greasy Gang&#39;]</span>
 <span class="c1"># 6.05s : name=&#39;Ice Cream Girl&#39; age=25 power=&#39;Can create ice cream out of thin air&#39; enemies=[&#39;The Hot Sauce Squad&#39;, &#39;The Healthy Eaters&#39;]</span>
 </code></pre></div>
+<h2 id="streamedresponse">StreamedResponse<a class="headerlink" href="#streamedresponse" title="Permanent link">&para;</a></h2>
+<p>Some LLMs have the ability to generate text output and make tool calls in the same response. This allows them to perform chain-of-thought reasoning or provide additional context to the user. In magentic, the <code>StreamedResponse</code> (or <code>AsyncStreamedResponse</code>) class can be used to request this type of output. This object is an iterable of <code>StreamedStr</code> (or <code>AsyncStreamedStr</code>) and <code>FunctionCall</code> instances.</p>
+<div class="admonition warning">
+<p class="admonition-title">Consuming StreamedStr</p>
+<p>The StreamedStr object must be iterated over before the next item in the <code>StreamedResponse</code> is processed, otherwise the string output will be lost. This is because the <code>StreamedResponse</code> and <code>StreamedStr</code> share the same underlying generator, so advancing the <code>StreamedResponse</code> iterator skips over the <code>StreamedStr</code> items. The <code>StreamedStr</code> object has internal caching so after iterating over it once the chunks will remain available.</p>
+</div>
+<p>In the example below, we request that the LLM generates a greeting and then calls a function to get the weather for two cities. The <code>StreamedResponse</code> object is then iterated over to print the output, and the <code>StreamedStr</code> and <code>FunctionCall</code> items are processed separately.</p>
+<div class="language-python highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">magentic</span> <span class="kn">import</span> <span class="n">prompt</span><span class="p">,</span> <span class="n">FunctionCall</span><span class="p">,</span> <span class="n">StreamedResponse</span><span class="p">,</span> <span class="n">StreamedStr</span>
+
+
+<span class="k">def</span> <span class="nf">get_weather</span><span class="p">(</span><span class="n">city</span><span class="p">:</span> <span class="nb">str</span><span class="p">)</span> <span class="o">-&gt;</span> <span class="nb">str</span><span class="p">:</span>
+    <span class="k">return</span> <span class="sa">f</span><span class="s2">&quot;The weather in </span><span class="si">{</span><span class="n">city</span><span class="si">}</span><span class="s2"> is 20°C.&quot;</span>
+
+
+<span class="nd">@prompt</span><span class="p">(</span>
+    <span class="s2">&quot;Say hello, then get the weather for: </span><span class="si">{cities}</span><span class="s2">&quot;</span><span class="p">,</span>
+    <span class="n">functions</span><span class="o">=</span><span class="p">[</span><span class="n">get_weather</span><span class="p">],</span>
+<span class="p">)</span>
+<span class="k">def</span> <span class="nf">describe_weather</span><span class="p">(</span><span class="n">cities</span><span class="p">:</span> <span class="nb">list</span><span class="p">[</span><span class="nb">str</span><span class="p">])</span> <span class="o">-&gt;</span> <span class="n">StreamedResponse</span><span class="p">:</span> <span class="o">...</span>
+
+
+<span class="n">response</span> <span class="o">=</span> <span class="n">describe_weather</span><span class="p">([</span><span class="s2">&quot;Cape Town&quot;</span><span class="p">,</span> <span class="s2">&quot;San Francisco&quot;</span><span class="p">])</span>
+<span class="k">for</span> <span class="n">item</span> <span class="ow">in</span> <span class="n">response</span><span class="p">:</span>
+    <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">item</span><span class="p">,</span> <span class="n">StreamedStr</span><span class="p">):</span>
+        <span class="k">for</span> <span class="n">chunk</span> <span class="ow">in</span> <span class="n">item</span><span class="p">:</span>
+            <span class="c1"># print the chunks as they are received</span>
+            <span class="nb">print</span><span class="p">(</span><span class="n">chunk</span><span class="p">,</span> <span class="n">sep</span><span class="o">=</span><span class="s2">&quot;&quot;</span><span class="p">,</span> <span class="n">end</span><span class="o">=</span><span class="s2">&quot;&quot;</span><span class="p">)</span>
+        <span class="nb">print</span><span class="p">()</span>
+    <span class="k">if</span> <span class="nb">isinstance</span><span class="p">(</span><span class="n">item</span><span class="p">,</span> <span class="n">FunctionCall</span><span class="p">):</span>
+        <span class="c1"># print the function call, then call it and print the result</span>
+        <span class="nb">print</span><span class="p">(</span><span class="n">item</span><span class="p">)</span>
+        <span class="nb">print</span><span class="p">(</span><span class="n">item</span><span class="p">())</span>
+
+<span class="c1"># Hello! I&#39;ll get the weather for Cape Town and San Francisco for you.</span>
+<span class="c1"># FunctionCall(&lt;function get_weather at 0x1109825c0&gt;, &#39;Cape Town&#39;)</span>
+<span class="c1"># The weather in Cape Town is 20°C.</span>
+<span class="c1"># FunctionCall(&lt;function get_weather at 0x1109825c0&gt;, &#39;San Francisco&#39;)</span>
+<span class="c1"># The weather in San Francisco is 20°C.</span>
+</code></pre></div>
 
 
 

diff --git a/structured-outputs/index.html b/structured-outputs/index.html
@@ -1007,6 +1007,10 @@ <h2 id="python-types">Python Types<a class="headerlink" href="#python-types" tit
 <span class="c1"># &gt; True</span>
 </code></pre></div>
 <h2 id="chain-of-thought-prompting">Chain-of-Thought Prompting<a class="headerlink" href="#chain-of-thought-prompting" title="Permanent link">&para;</a></h2>
+<div class="admonition warning">
+<p class="admonition-title">StreamedResponse</p>
+<p>It is now recommended to use <code>StreamedResponse</code> for chain-of-thought prompting, as this uses the LLM provider's native chain-of-thought capabilities. See <a href="../streaming/#StreamedResponse">StreamedResponse</a> for more information.</p>
+</div>
 <p>Using a simple Python type as the return annotation might result in poor results as the LLM has no time to arrange its thoughts before answering. To allow the LLM to work through this "chain of thought" you can instead return a pydantic model with initial fields for explaining the final response.</p>
 <div class="language-python highlight"><pre><span></span><code><span class="kn">from</span> <span class="nn">magentic</span> <span class="kn">import</span> <span class="n">prompt</span>
 <span class="kn">from</span> <span class="nn">pydantic</span> <span class="kn">import</span> <span class="n">BaseModel</span><span class="p">,</span> <span class="n">Field</span>