We introduce OctoTools, a training-free, user-friendly, and easily extensible open-source agentic framework designed to tackle complex reasoning across diverse domains. OctoTools introduces standardized tool cards to encapsulate tool functionality, a planner for both high-level and low-level planning, and an executor to carry out tool usage.
Tool cards define tool-usage metadata and encapsulate heterogeneous tools, enabling training-free integration of new tools without additional training or framework refinement. (2) The planner governs both high-level and low-level planning to address the global objective and refine actions step by step. (3) The executor instantiates tool calls by generating executable commands and save structured results in the context. The final answer is summarized from the full trajectory in the context. Furthermore, the task-specific toolset optimization algorithm learns a beneficial subset of tools for downstream tasks.
We validate OctoTools’ generality across 16 diverse tasks (including MathVista, MMLU-Pro, MedQA, and GAIA-Text), achieving substantial average accuracy gains of 9.3% over GPT-4o. Furthermore, OctoTools also outperforms AutoGen, GPT-Functions and LangChain by up to 10.6% when given the same set of tools.
Create a conda environment from the conda.yaml file:
conda env create -f conda.yamlActivate the environment and install requirements:
conda activate octotools
pip install -e .Make .env file, and set OPENAI_API_KEY, GOOGLE_API_KEY, GOOGLE_CX, etc. For example:
# The content of the .env file
# Used for GPT-4o-powered tools
OPENAI_API_KEY=<your-api-key-here>
# Used for the Google Search tool
GOOGLE_API_KEY=<your-api-key-here>
GOOGLE_CX=<your-cx-here>
# Used for the Advanced Object Detector tool (Optional)
DINO_KEY=<your-dino-key-here>Obtain a Google API Key and Google CX according to the Google Custom Search API documation.
Install parallel for running benchmark experiments in parallel:
sudo apt-get update
sudo apt-get install parallelUsing Python_Code_Generator_Tool as an example, test the availability of the tool by running the following:
cd octotools/tools/python_code_generator
python tool.pyExpected output:
Execution Result: {'printed_output': 'The sum of all the numbers in the list is: 15', 'variables': {'numbers': [1, 2, 3, 4, 5], 'total_sum': 15}}
You can also test all tools available in the toolbox by running the following:
cd octotools/tools
source test_all_tools.shExpected testing log:
Testing advanced_object_detector...
✅ advanced_object_detector passed
Testing arxiv_paper_searcher...
✅ arxiv_paper_searcher passed
...
Testing wikipedia_knowledge_searcher...
✅ wikipedia_knowledge_searcher passed
Done testing all tools
Failed: 0
Using CLEVR-Math as an example, run inference on a benchmark by:
cd octotools/tasks
# Run inference from clevr-math using GPT-4 only
source clevr-math/run_gpt4o.sh
# Run inference from clevr-math using the base tool
source clevr-math/run_octotool_base.sh
# Run inference from clevr-math using Octotools with an optimized toolset
source clevr-math/run_octotools.shMore benchmarks are available in the tasks.
To demonstrate the generality of our OctoTools framework, we conduct comprehensive evaluations on 16 diverse benchmarks spanning two modalities, five domains, and four reasoning types. These benchmarks encompass a wide range of complex reasoning tasks, including visual understanding, numerical calculation, knowledge retrieval, and multi-step reasoning.
More results are available in the paper or at the project page.
We provide a set of in-depth analyses to help you understand the framework. For instance, we visualize the tool usage of OctoTools and its baselines from 16 tasks. It turns out that OctoTools takes advantage of different external tools to address task-specific challenges. Explore more findings at our paper or the project page.
We provide a set of example visualizations to help you understand the framework. Explore them at the project page.
The design of each tool card is modular relative to the OctoTools framework, enabling users to integrate diverse tools without modifying the underlying framework or agent logic. New tool cards can be added, replaced, or updated with minimal effort, making OctoTools robust and extensible as tasks grow in complexity.
To customize OctoTools for your own tasks:
-
Add a new tool card: Implement your tool following the structure in existing tools.
-
Replace or update existing tools: You can replace or update tools in the toolbox. For example, we provide the
Object_Detector_Toolto detect objects in images using an open-source model. We also provide an alternative tool called theAdvanced_Object_Detector_Toolto detect objects in images using API calls. -
Enable tools for your tasks: You can enable the whole toolset or a subset of tools for your own tasks by setting the
enabled_toolsargument in tasks/solve.py.
This project draws inspiration from several remarkable projects:
- 📕 Chameleon – Chameleon is an early attempt that augments LLMs with tools, which is a major source of inspiration. A journey of a thousand miles begins with a single step.
- 📘 TextGrad – We admire and appreciate TextGrad for its innovative and elegant framework design.
- 📗 AutoGen – A trending project that excels in building agentic systems.
- 📙 LangChain – A powerful framework for constructing agentic systems, known for its rich functionalities.
@article{lu2025octotools,
title={OctoTools: An Agentic Framework with Extensible Tools for Complex Reasoning},
author={Lu, Pan and Chen, Bowen and Liu, Sheng and Thapa, Rahul and Boen, Joseph and Zou, James},
journal = {arXiv preprint arXiv:2502.11271},
year={2025}
}
Pan Lu |
Bowen Chen |
Sheng Liu |
Rahul Thapa |
)
Joseph Boen |
James Zou |
We are trully looking forward to the open-source contributions to OctoTools! If you are interested in contributing, collaborating, or reporting issues, don't hesitate to contact us!
We are also looking forward to your feedback and suggestions!