Persistent Tools Library and Dynamic Tool Loader

[streichsbaer]

We already have great contributions related to tool making and tool use, such as:

• Auto Tool Making and External Interaction #88
• Prototype for Communication and Collaboration over GitHub #113

The work on the Unit Manager by @OWigginsHay is very helpful in building new tools/functions and passing it to an Agent during instantiation. In addition, purposefully creating certain tools that are part of the initial/core toolbox, such as the GitHub API Wrapper Prototype by @RomanGoEmpire has a lot of value.

I would like to explore ideas around creating a Persistent Tool Library (PTL) and how to make tools accessible to the Swarm.

These are my thoughts so far:

1. The Persistent Tool Library needs CRUD (create, read, update, delete) abilities on tools to allow continuous self-improvement and management of available tools.
2. Tools may need to be grouped together, e.g. the GitHub API Wrapper class is a multi-tool with the ability to fully interact with the GitHub API.
3. The Dynamic Tool Loader (DTL) helps provide agents with the right tools at instantiation time to achieve their goals and not waste tokens/context on irrelevant tool definitions.
4. There can be stateful agents who are continuously evolving and are experts on certain tasks and the usage of certain tools. Stateful agents could also self-organize into agent teams.
5. There can also be ephemeral agents that just run a single tool/function.

There are two papers I found very helpful for this specific topic:

1. Voyager: An Open-Ended Embodied Agent with Large Language Models that is able to autonomously build an ever-growing skill library and can incorporate environment feedback, execution errors, and self-verification for program improvements. This is useful to create the skills that are available for ephemeral agents, but also for stateful expert agents.
2. Generative Agents: Interactive Simulacra of Human Behavior describes an architecture that extends a large language model to store a complete record of agent's experiences using natural language, synthesize those memories over time into higher-level reflections, and retrieve them dynamically to plan behavior. This is useful if the concept of stateful, self-improving agents makes sense for the Swarm architecture.

What do we think about this, dear collective?

[RomanGoEmpire]

Great summary and thoughts with the additions of the papers!
If you connect the Tool-maker with GitHub you will be able to create and work on tools iteratively while saving the progress.

3. The Dynamic Tool Loader (DTL) helps provide agents with the right tools at instantiation time to achieve their goals and not waste tokens/context on irrelevant tool definitions.

This would be similar to the two assistants we already have for answering questions about this repo.

Like this the Tools will be in one place and synchronized for all other agents.

[mikerhodesideas]

Given the newish direction to use github as the core 'brain' for discussions, I really like the idea of using a wiki to store all the data about the tools too.
When teaching humans about this, I find THE best place to start is an overview of "what's possible" - I think the PTL serves the same purpose for the swarm.
do we need to describe an internal schema for tools? is that overkill?

eg,

• what level the tool can be used by (tier 5 probably won't have access to everything)
• maybe existing 'skills' needed in order to consider using the tool (riffing off the voyager paper - a tech tree of sorts)
• really good names, descriptions (maybe one for the API & one for humans?)
• a log of updates (who, when, why)
• the class/subclass of tools it belongs to (grouping together makes searching easier - esp for human oversight!)
• maybe a log of the runs in which it's been used (for random checks, improvement, roll backs?)

I'm sure there's more...

[OWigginsHay]

@mikerhodesideas I started writing up a introduction to the tools work in the wiki last night but got too tired before sleeping. I aim to update it today, cause there are many ideas going around and I want to integrate everything in an approach that fits with the already delivered implementations too.

I think a schema is absolutely needed. Few reasons why: When we call functions, we need some way to represent the input arguments to that function, this is usually a dict also representative as a JSON object. If not forced, assistant may try to call functions using natural language, lists, code blocks, markdown, or json but that is a different function every time. Having a standardized json schema will help implement and reuse code to check the validity of outputs and feedback to assistant when they are hallucinating results or creating missinput. The other reason is if all agents share a schema and agents have group chats and methods for self monitoring and self improvements, all agents will very quickly learn how to abide by the schema and not make up their own. It helps us too as programmers where that schema can be an abstract class, any time we pull a function, tool request or message gong between two targets, we know ti AT LEAST has all the attributes of our abstract schema.

Agree with many of your other point here, logs are good and I really want persistent threads and histories etc too

[OWigginsHay]

Great summary, I am writing Tools Work and Milestones on the wiki now. With Tools, there are two types we need to start considering and differentiating. Assistant tools that are like python functions generated and used by working assistant upon request (potentially shared with other agents), and Swarm tools which are more like external tools we have built as programmers that we give agents an interface to talk with.

Agree with assessment criteria to decide which set of default tools agents begin with.

Now to the most important idea I am going to start trying to introduce is "Extensions". As a high level, the Functional Assistant and Thread in my unit is an Extension, it's a programming Extension. It's a slave assistant locked in a thread with schema input that produces output which either is returned to the Assistant or the larger system. Other extension could exist such also such as a Ethics Compliance Extension, Requirements Enforcement Extension etc etc

[OWigginsHay]

#123

[daveshap]

I think a Tool Library is a critical step for Tool Making (and Refinement) e.g. each tool has a UUID and we can keep a copy of each version to characterize it's success and failure. Make it so.

[ph47s74x]

This project has a good approach to the organization and creation of tools for modularity.

https://github.com/griptape-ai/griptape/tree/dev/griptape/tools

any thoughts?