diff --git a/docs/_static/chatbot.png b/docs/_static/chatbot.png new file mode 100644 index 00000000..c742c2d2 Binary files /dev/null and b/docs/_static/chatbot.png differ diff --git a/docs/_static/counter.png b/docs/_static/counter.png deleted file mode 100644 index db6bc637..00000000 Binary files a/docs/_static/counter.png and /dev/null differ diff --git a/docs/concepts/actions.rst b/docs/concepts/actions.rst index 02a8d08f..0f74b1d6 100644 --- a/docs/concepts/actions.rst +++ b/docs/concepts/actions.rst @@ -4,6 +4,12 @@ Actions .. _actions: +.. note:: + + Actions are the core building block of Burr. They read from state and write to state. + They can be synchronous and asynchonous, and have both a ``sync`` and ``async`` API. + There are both function and class-based APIs. + Actions do the heavy-lifting in a workflow. They should contain all complex compute. You can define actions either through a class-based or function-based API. If actions implement ``async def run`` then will be run in an diff --git a/docs/concepts/additional-visibility.rst b/docs/concepts/additional-visibility.rst index 9a6fa14d..20f69488 100644 --- a/docs/concepts/additional-visibility.rst +++ b/docs/concepts/additional-visibility.rst @@ -2,9 +2,10 @@ Additional Visibility ===================== -Burr comes with the ability to see inside your actions. This is a very pluggable framework -that comes with the default tracking client. +.. note:: + Burr comes with the ability to see inside your actions. This is a very pluggable framework + that comes with the default tracking client, but can also be hooked up to tools such as `OpenTelemetry `_ ------- Tracing diff --git a/docs/concepts/hooks.rst b/docs/concepts/hooks.rst index 3708573b..8ff6100c 100644 --- a/docs/concepts/hooks.rst +++ b/docs/concepts/hooks.rst @@ -3,6 +3,10 @@ Hooks ===== .. _hooks: +.. note:: + + Hooks allow you to customize every aspect of Burr's execution, plugging in whatever tooling, + observability framework, or debugging tool you need. Burr has a system of lifecycle adapters (adapted from the similar `Hamilton `_ concept), which allow you to run tooling before and after various places in a node's execution. For instance, you could: diff --git a/docs/concepts/index.rst b/docs/concepts/index.rst index 1458d5ef..7b5be408 100644 --- a/docs/concepts/index.rst +++ b/docs/concepts/index.rst @@ -9,13 +9,13 @@ Overview of the concepts -- read these to get a mental model for how Burr works. .. toctree:: :maxdepth: 2 - state-machine - state actions + state + state-machine transitions - hooks tracking state-persistence streaming-actions + hooks additional-visibility planned-capabilities diff --git a/docs/concepts/state-machine.rst b/docs/concepts/state-machine.rst index 4101dcd0..f6619bec 100644 --- a/docs/concepts/state-machine.rst +++ b/docs/concepts/state-machine.rst @@ -4,6 +4,11 @@ Applications .. _applications: +.. note:: + + Applications tie everything in Burr together. They specify which actions should be included, + how they connect (including conditional edges), and how to run. + Applications form the core representation of the state machine. You build them with the ``ApplicationBuilder``. Here is the minimum that is required: @@ -18,18 +23,15 @@ This is shown in the example from :ref:`getting started ` from burr.core import ApplicationBuilder, default, expr app = ( ApplicationBuilder() - .with_state(counter=0) # initialize the count to zero - .with_actions( - count=count, # add the counter action with the name "counter" - done=done # add the printer action with the name "printer" - ).with_transitions( - ("count", "count", expr("counter < 10")), # Keep counting if the counter is less than 10 - ("count", "done", default) # Otherwise, we're done - ).with_entrypoint("counter") # we have to start somewhere + .with_actions(human_input, ai_response) + .with_transitions( + ("human_input", "ai_response"), + ("ai_response", "human_input") + ).with_state(chat_history=[]) + .with_entrypoint("human_input") .build() ) - ------- Running ------- diff --git a/docs/concepts/state-persistence.rst b/docs/concepts/state-persistence.rst index f31b708b..c12ba29f 100644 --- a/docs/concepts/state-persistence.rst +++ b/docs/concepts/state-persistence.rst @@ -4,6 +4,13 @@ State Persistence .. _state-persistence: +.. note:: + + Burr comes with a core set of APIs that enable state `persistence` -- the ability + to save and load state automatically from a database. This enables you to launch an application, + pause it, and restart where you left off. The API is customizable, and works with any database you want. + + The key to writing a real life ``burr`` application is state persistence. For example, say you're building a chat bot and you want to store the conversation history and then reload it when you restart. Or, you have a long running process/series of agents, and you want to store the state of the process after each action, and then reload it if it fails, etc.... diff --git a/docs/concepts/state.rst b/docs/concepts/state.rst index cbd1bc69..883e2348 100644 --- a/docs/concepts/state.rst +++ b/docs/concepts/state.rst @@ -4,6 +4,11 @@ State .. _state: +.. note:: + + Burr's ``State`` API enables actions to talk to each other, and enables you to persist data. + Burr has a ``State`` API that allows you to manipulate state in a functional way. + The ``State`` class provides the ability to manipulate state for a given action. It is entirely immutable, meaning that you can only create new states from old ones, not modify them in place. diff --git a/docs/concepts/streaming-actions.rst b/docs/concepts/streaming-actions.rst index e8de5d88..81736cb7 100644 --- a/docs/concepts/streaming-actions.rst +++ b/docs/concepts/streaming-actions.rst @@ -4,6 +4,11 @@ Streaming Actions .. _streaming_actions: +.. note:: + + Burr actions can stream results! This enables you to display results to the user as + tokens are streamed in. + Actions can be implemented as streaming results. This enables a lower time-to-first-token and a more interactive interface in the case of AI applications or streaming in of metrics in a model-training application. Broadly, this is a tool to enable quicker user interaction in longer running actions that require user focus. diff --git a/docs/concepts/tracking.rst b/docs/concepts/tracking.rst index ca1dcff5..f3be5a7f 100644 --- a/docs/concepts/tracking.rst +++ b/docs/concepts/tracking.rst @@ -4,6 +4,12 @@ Tracking Burr ============= +.. note:: + + Burr's telemetry system is built in and easy to integrate. It allows you to understand + the flow of your application, and watch it make decisions in real time. You can run it + with sample projects by running ``burr`` in the terminal after ``pip install "burr[start]"``. + Burr comes with a telemetry system that allows tracking a variety of information for debugging, both in development and production. diff --git a/docs/concepts/transitions.rst b/docs/concepts/transitions.rst index 4f304729..ec79ca03 100644 --- a/docs/concepts/transitions.rst +++ b/docs/concepts/transitions.rst @@ -4,6 +4,12 @@ Transitions .. _transitions: +.. note:: + + While actions form the steps taken in an `application`, transitions decide which one to do next. + They make decisions based on state. You can use them to specify which model to call, whether a conversation is + over, or any other decision that needs to be made based on the current state. + Transitions define explicitly how actions are connected and which action is available next for any given state. You can think of them as edges in a graph. diff --git a/docs/getting_started/install.rst b/docs/getting_started/install.rst index 5095ddad..fa7c8528 100644 --- a/docs/getting_started/install.rst +++ b/docs/getting_started/install.rst @@ -9,6 +9,7 @@ along with a fully built server. .. code-block:: bash - pip install burr[start] + pip install "burr[start]" -This will give you tools to visualize, track, and interact with the UI. Note, if you're using ``zsh``, you'll need to add quotes around the install target, (``pip install "burr[start]"``). +This will give you tools to visualize, track, and interact with the UI. You can explore the UI (including some sample projects) +simply by running the command ``burr``. Up next we'll write our own application and follow it in the UI. diff --git a/docs/getting_started/simple-example.rst b/docs/getting_started/simple-example.rst index f6ae9ba3..90b3c447 100644 --- a/docs/getting_started/simple-example.rst +++ b/docs/getting_started/simple-example.rst @@ -1,90 +1,139 @@ .. _simpleexample: -================= +============== Simple Example -================= -This simple example is just to learn the basics of the library. The application we're building is not particularly interesting, -but it will get you powerful. If you want to skip ahead to the cool stuff (chatbots, -ML training, simulations, etc...) feel free to jump into the deep end and start with the :ref:`examples `. +============== -We will go over enough of the concepts to help you understand the code, but there is a much more in-depth set of -explanations in the :ref:`concepts ` section. +Let's build going to build a basic chatbot using Burr. While Burr has a host of features related to state management and inspection, +this basic tutorial is going to demonstrate two that are particularly relevant to LLM apps. -🤔 For an alternative example to get started with, you can also check out +1. Specifying user inputs +2. Persisting state across multiple interactions + +This tutorial will set you up to do more complex things, including: + +1. Building an application that makes decisions/handles conditional edges +2. Persisting state in a database +3. Tracking/monitoring your application +4. Generating test cases from prior application runs + +So hold tight! This gets you started with the basics but there's a lot more you can do with little effort. + + +.. note:: + + This should take about 10 minutes to complete, and give you a good sense of the library basics. + You'll need an OpenAI key set as the environment variable ``OPENAI_API_KEY``. If you don't have one you can get one at `OpenAI `_. + If you don't want to get one, check out the simple example of a `counter application `_. + + If you want to skip ahead to the cool stuff (chatbots, ML training, simulations, etc...) feel free to jump into the deep end and start with the :ref:`examples `. + +🤔 If you prefer to learn by video, check out `this video walkthrough `_ -using `this notebook `_. +using `this notebook `_. ------------------- -Build a Counter ------------------- -We're going to build a counter application. This counter will count to 10 and then stop. +---------------------- +Build a Simple Chatbot +---------------------- + +Let's build! Our chatbot will accept user input and pass it to an AI. The AI will then respond with a message. Let's start by defining some actions, the building-block of Burr. You can think of actions as a function that computes a result and modifies state. They declare what they read and write. -Let's define two actions: -1. A counter action that increments the counter -2. A printer action that prints the counter + +We define two actions: + +1. ``human_input`` This accepts a prompt from the outside and adds it to the state +2. ``ai_response`` This takes the prompt + chat history and queries OpenAI. .. code-block:: python - @action(reads=["counter"], writes=["counter"]) - def count(state: State) -> Tuple[dict, State]: - current = state["counter"] + 1 - result = {"counter": current} - print("counted to ", current) - return result, state.update(**result) # return both the intermediate + client = openai.Client() + + @action(reads=[], writes=["prompt", "chat_history"]) + def human_input(state: State, prompt: str) -> Tuple[dict, State]: + chat_item = { + "content": prompt, + "role": "user" + } + return ( + {"prompt": prompt}, + state.update(prompt=prompt).append(chat_history=chat_item) + ) - @action(reads=["counter"], writes=[]) - def done(state: State) -> Tuple[dict, State]: - print("Bob's your uncle") - return {}, state + @action(reads=["chat_history"], writes=["response", "chat_history"]) + def ai_response(state: State) -> Tuple[dict, State]: + content = client.chat.completions.create( + model="gpt-3.5-turbo", + messages=state["chat_history"], + ).choices[0].message.content + chat_item = { + "content": content, + "role": "assistant" + } + return ( + {"response": content}, + state.update(response=content).append(chat_history=chat_item) + ) + +Before we proceed, let's note the following about how we define these actions: -This is an action that reads the "counter" from the state, increments it by 1, and then writes the new value back to the state. -It returns both the intermediate result (``result``) and the updated state. +1. State is a dictionary -- actions declare input fields (as strings) and write values to those fields +2. Actions use a specific *immutable* state object and call operations on it (``.append(...)``, ``.update(...)``) +3. Functions can do whatever you want -- they can use plain python, or delegate to `langchain `_, `hamilton `_, etc... +4. We declare the parameter ``prompt``, meaning that we will expect the user to pass ``prompt`` every time they run the graph. -Next, let's put together our application. To do this, we'll use an ``ApplicationBuilder`` +Next, let's piece together our application. To do this, we'll use an ``ApplicationBuilder`` .. code-block:: python - from burr.core import ApplicationBuilder, default, expr app = ( ApplicationBuilder() - .with_state(counter=0) # initialize the count to zero - .with_actions( - count=count, # add the counter action with the name "counter" - done=done # add the printer action with the name "printer" - ).with_transitions( - ("count", "count", expr("counter < 10")), # Keep counting if the counter is less than 10 - ("count", "done", default) # Otherwise, we're done - ).with_entrypoint("count") # we have to start somewhere - .with_tracker(project="my_first_app") + .with_actions(human_input, ai_response) + .with_transitions( + ("human_input", "ai_response"), + ("ai_response", "human_input") + ).with_state(chat_history=[]) + .with_entrypoint("human_input") .build() ) - We can visualize the application (note you need ``burr[graphviz]`` installed): .. code-block:: python - app.visualize("./graph", format="png", include_conditions=True, include_state=True) + app.visualize("./graph", format="png") -.. image:: ../_static/counter.png +.. image:: ../_static/chatbot.png :align: center -As you can see, we have: +Let's note the following about how we define the application -1. The action ``count`` that reads and writes the ``counter`` state field -2. The action ``done`` that reads the ``counter`` state field -3. A transition from ``count`` to ``count`` if ``counter < 10`` -4. A transition from ``count`` to ``done`` otherwise +1. It is an infinite loop! It is meant to pause for new prompt input. +2. We're just using the function names as the action names. You can also name them if you want ``with_actions(human_input=human_input, ai_response=ai_response)``. +3. We start it with an empty ``chat_history`` state field +4. It utilizes a `builder pattern `_ -- this is a bit old-school (comes from the java days), but is an easy/modular way to express your appliaction -Finally, we can run the application: +Finally, we can run the application -- it gives back multiple pieces of information but all we'll use is the state. .. code-block:: python - app.run(halt_after=["printer"]) + *_, state = app.run(halt_after=["ai_response"], inputs={"prompt": "Who was Aaron Burr?"}) + print("answer:", app.state["response"]) + print(len(state["chat_history"]), "items in chat") + +The result looks exactly as you'd expect! + +.. code-block:: text + + answer: Aaron Burr was an American politician and lawyer who served as the third + Vice President of the United States from 1801 to 1805 under President Thomas Jefferson. + He is also known for killing Alexander Hamilton in a famous duel in 1804. + Burr was involved in various political intrigues and controversies throughout his career, + and his reputation was tarnished by suspicions of treason and conspiracy. + 2 items in chat If you want to copy/paste, you can open up the following code block and add to a file called ``run.py``: @@ -92,66 +141,55 @@ If you want to copy/paste, you can open up the following code block and add to a .. code-block:: python + import uuid from typing import Tuple - from burr.core import ( - action, - State, - ApplicationBuilder, - default, - expr - ) - - - @action(reads=["counter"], writes=["counter"]) - def count(state: State) -> Tuple[dict, State]: - current = state["counter"] + 1 - result = {"counter": current} - print("counted to ", current) - return result, state.update(**result) # return both the intermediate - - - @action(reads=["counter"], writes=[]) - def done(state: State) -> Tuple[dict, State]: - print("Bob's your uncle") - return {}, state - - - if __name__ == '__main__': - app = ( - ApplicationBuilder() - .with_state(counter=0) # initialize the count to zero - .with_actions( - count=count, # add the counter action with the name "counter" - done=done # add the printer action with the name "printer" - ).with_transitions( - ("count", "count", expr("counter < 10")), # Keep counting if the counter is less than 10 - ("count", "done", default) # Otherwise, we're done - ).with_entrypoint("count") # we have to start somewhere - .with_tracker(project="my_first_app") - .build() - ) - app.visualize("./graph", format="png", include_conditions=True, include_state=True) - app.run(halt_after=["done"]) + import openai + from burr.core import action, State, ApplicationBuilder, when, persistence -And the output looks exactly as we expect! + client = openai.Client() -.. code-block:: text + @action(reads=[], writes=["prompt", "chat_history"]) + def human_input(state: State, prompt: str) -> Tuple[dict, State]: + chat_item = { + "content": prompt, + "role": "user" + } + return ( + {"prompt": prompt}, + state.update(prompt=prompt).append(chat_history=chat_item) + ) - $ python run.py + @action(reads=["chat_history"], writes=["response", "chat_history"]) + def ai_response(state: State) -> Tuple[dict, State]: + content = client.chat.completions.create( + model="gpt-3.5-turbo", + messages=state["chat_history"], + ).choices[0].message.content + chat_item = { + "content": content, + "role": "assistant" + } + return ( + {"response": content}, + state.update(response=content).append(chat_history=chat_item) + ) + app = ( + ApplicationBuilder() + .with_actions(human_input, ai_response) + .with_transitions( + ("human_input", "ai_response"), + ("ai_response", "human_input") + ).with_state(chat_history=[]) + .with_entrypoint("human_input") + .build() + ) + app.visualize("./graph", format="png") + *_, state = app.run(halt_after=["ai_response"], inputs={"prompt": "Who was Aaron Burr?"}) + print("answer:", app.state["response"]) + print(len(state["chat_history"]), "items in chat") - counted to 1 - counted to 2 - counted to 3 - counted to 4 - counted to 5 - counted to 6 - counted to 7 - counted to 8 - counted to 9 - counted to 10 - Bob's your uncle Finally, let's open up the UI and see what it looks like (note, that if you have not installed ``burr[learn]`` now is a good time...). @@ -161,5 +199,10 @@ Finally, let's open up the UI and see what it looks like (note, that if you have You'll see the UI pop up with projects. Navigate to `the UI `_ and explore! -All this to increment? Well, if all you want to do is count to 10, this might not be for you. But we imagine most of you want to do more exciting things -than count to 10... + +Now that we've built a basic application, we can do the following with only a few lines of code: + +1. :ref:`Add conditional edges ` -- add a condition as a third item in the tuple to the ``with_transitions`` method. +2. :ref:`Persist state to a database + reload ` -- add a ``initialize_from`` line to the builder and select a pre-existing/implement a custom persistence method. +3. :ref:`Add monitoring to track application data ` -- leverage ``with_tracker`` to track to the Burr UI and visualize your application live. +4. `Generate test cases from prior runs `_ -- use the ``burr-testburr-test-case create`` command to automatically generate test cases for your LLM app. diff --git a/docs/getting_started/why-burr.rst b/docs/getting_started/why-burr.rst index be48ae7c..b8a02639 100644 --- a/docs/getting_started/why-burr.rst +++ b/docs/getting_started/why-burr.rst @@ -1,50 +1,21 @@ -================= -Why Burr -================= - ---------------------------------------- -An abstraction to make your life easier ---------------------------------------- +========= +Why Burr? +========= Why do you need a state machine for your applications? Won't the normal programming constructs suffice? -Yes, until a point. Let's look at a chatbot as an example. Here's a simple design of something gpt-like: - -#. Accept a prompt from the user -#. Perform some simple checks/validations on that prompt (is it safe/within the terms of service) -#. If (2) then decide the mode to which to respond to that prompt from a set of capabilities. Else responds accordingly: - * Generate an image - * Answer a question - * Write some code - * ... -#. Query the appropriate model with the prompt, formatted as expected - * On failure, present an error message - * On success, present the response to the user -#. Await a new prompt, GOTO (1) - -Visually, we might have an implementation/spec that looks like this: - -.. image:: ../_static/demo_graph.png - :align: center - - -While this involves multiple API calls, error-handling, etc... it is definitely possible to get a prototype -that looks slick out without too much abstraction. - -Now, let's get this to production. We need to: +**Yes, until a point.** Let's take a look at what you need to build a production-level LLM application: -#. Add monitoring to figure out if/why any of our API calls return strange results -#. Understand the decisions made by the application -- E.G. why it chose certain modes, why it formatted a response correctly. This involves: - * Tracking all the prompts/responses - * Going back in time to examine the state of the application at a given point -#. Debug it in a local mode, step-by-step, using the state we observed in production -#. Add new capabilities to the application -#. Monitor the performance of the application -- which steps/decisions are taking the longest? -#. Monitor the cost of running the application -- how many tokens are we accepting from/delivering to the user. -#. Save the state out to some external store so you can restart the conversation from where you left off +1. **Tracing/telemetry** -- LLMs can be chaotic, and you need visibility into what decisions it made and how long it took to make them. +2. **State persistence** -- thinking about how to save/load your application is a whole other level of infrastructure you need to worry about. +3. **Visualization/debugging** -- when developing you'll want to be able to view what it is doing/did + load up the data at any point +4. **Manage interaction between users/LLM** -- pause for input in certain conditions +5. **Data gathering for evaluation + test generation** -- storing data run in production to use for later analysis/fine-tuning -And this is the tip of the iceberg -- chatbots (and all stateful applications) get really complicated, really quickly. +You can always patch together various frameworks or build it all yourself, but at that point you're going to be spending a lot of time on tasks that +are not related to the core value proposition of your software. -Burr is designed to unlock the capabilities above and more -- decomposing your application into functions that manipulate state -and transition, with hooks that allow you to customize any part of the application. It is a platform on top of which you can build any of the -production requirements above, and comes with many of them out of the box! +Burr was built with this all in mind. By modeling your application as a state machine of simple python constructs you can have the best of both worlds. +Bring in whatever infrastructure/tooling you want and get all of the above. Burr is meant to start off as an extremely lightweight tool to +make building LLM (+ a wide swath of other) applications easier. The value compounds as you leverage more of the ecosystems, plugins, and additional +features it provides.