Support multiple tool registries #1193

njhale · 2025-01-09T22:27:44Z

This change adds support for consuming tools from more than one tool registry at a time.

It replaces the OBOT_SERVER_TOOL_REGISTRY option with OBOT_SERVER_TOOL_REGISTRIES. The new option accepts a comma-delimited string of registry URIs.

e.g.

export OBOT_SERVER_TOOL_REGISTRIES='github.com/obot-platform/tools,github.com/obot-platform/enterprise-tools'`

When overriding a remote registry with a directory in the local filesystem, the GPTSCRIPT_TOOL_REMAP environment variable is used to provide tool reference remapping in GPTScript/Obot.

e.g. Replacing github.com/obot-platform/tools with ../tools:

export GPTSCRIPT_TOOL_REMAP='github.com/obot-platform/tools=../tools'

e.g. Replacing github.com/obot-platform/tools with ../tools and github.com/obot-platform/enterprise-tools with ../enterprise-tools:

export GPTSCRIPT_TOOL_REMAP='github.com/obot-platform/tools=../tools,github.com/obot-platform/enterprise-tools=../enterprise-tools'
export OBOT_SERVER_TOOL_REGISTRIES='github.com/obot-platform/tools,github.com/obot-platform/enterprise-tools'`

Note:

OBOT_SERVER_TOOL_REGISTRIES should always reference remote tool registry repos, never local file paths
Only GPTSCRIPT_TOOL_REMAP needs to be set to swap the default tool registry (github.com/obot-platform/tools) with a local fork

This change also updates the container image build to support

Pulling multiple tool registries into the image at build-time via the TOOL_REGISTY_REPOS build arg
Cloning private tool registry repos via the GITHUB_TOKEN Docker secret

e.g. Building the "enterprise" Obot image

export GITHUB_TOKEN=$(gh auth token)
# Here `github.com/obot-platform/enterprise-tools` is a private repo
docker build --build-arg TOOL_REGISTRY_REPOS='github.com/obot-platform/enterprise-tools,github.com/obot-platform/tools' \
  --secret id=GITHUB_TOKEN \
  -t obot-enterprise:latest .

e.g. Building the "OSS" Obot image

# When no `TOOL_REGISTRY_REPOS` build arg is given, `github.com/obot-platform/obot` is pulled into the image by default
docker build -t obot:latest .

Part of #877

.github/workflows/docker-build-and-push.yml

tools/package-tools.sh

pkg/storage/server.go

njhale · 2025-01-15T08:38:57Z

Still need to test the workflow, but I think this is ready for some reviews.

See this commit message for an example of how to build the enterprise image locally. You'll need to tweak the TOOL_REGISTRY_REPO environment variable since the enterprise repo hasn't been bootstrapped yet. Try the command below to use my updated registry forks for now:

docker build --build-arg TOOL_REGISTRY_REPOS='github.com/njhale/enterprise-tools,github.com/njhale/tools' \
  --secret id=GITHUB_TOKEN \
  -t obot-enterprise:latest .

njhale · 2025-01-15T08:42:06Z

We'll also want to disable model providers and oauthapps in the UI when their tools aren't present. We don't want to show the options for the "OSS" build if you can't use them.

njhale · 2025-01-15T08:44:20Z

DEVELOPMENT.md

 1. Clone `github.com/obot-platform/tools` to your local machine.
 2. In the root directory of the tools repo on your local machine, run `make build`.
-3. Run the Obot server, either with `make dev` or in your IDE, with the `OBOT_SERVER_TOOL_REGISTRY` environment variable set to the root directory of the tools repo.
+3. Run the Obot server, either with `make dev` or in your IDE, with the `GPTSCRIPT_TOOL_REMAP` environment variable set to `github.com/obot-platform/tools=<local-tools-fork-root-directory>`; e.g. If you cloned the tools repo to the directory "above" the Obot repo, you'd use `GPTSCRIPT_TOOL_REMAP='github.com/obot-platform/tools=../tools' make dev`.

 Now, any time one of these tools is run, your local copy will be used.


Thinking about putting the docs for developing for "enterprise" in the enterprise-tools repo. LMK if folks would prefer it here in the obot repo instead.

njhale · 2025-01-15T08:46:56Z

pkg/credstores/credstore.go

@@ -88,7 +95,33 @@ func setupSQLite(toolRegistry, dsn string) (string, []string, error) {

 	dbFile = strings.TrimSuffix(dbFile, ".db") + "-credentials.db"

-	return toolRegistry + "/credential-stores/sqlite", []string{
+	toolRef, err := resolveToolRef(toolRegistries, "credential-stores/sqlite")


we do this because we're not guaranteed that the default tool registry is present in toolRegistries, so we make a best-effort attempt to find the required credential-stores tools in the available registries.

njhale · 2025-01-15T08:49:29Z

pkg/services/config.go

-	WorkspaceTool              string   `usage:"The tool reference for the workspace provider" default:"github.com/gptscript-ai/workspace-provider"`
-	DatasetsTool               string   `usage:"The tool reference for the dataset provider" default:"github.com/gptscript-ai/datasets"`


These are being removed because they can now be remapped directly with the GPTSCRIPT_TOOL_REMAP environment variable; e.g. export GPTSCRIPT_TOOL_REMAP=github.com/gptscript-ai/workspace-provider=../tools/workspace-provider

njhale · 2025-01-15T08:51:29Z

pkg/services/config.go

@@ -201,7 +194,11 @@ func New(ctx context.Context, config Config) (*Services, error) {
 	// that use postgres
 	config.DSN = strings.Replace(config.DSN, "postgresql://", "postgres://", 1)

-	credStore, credStoreEnv, err := credstores.Init(ctx, config.ToolRegistry, config.DSN, credstores.Options{
+	if len(config.ToolRegistries) < 1 {
+		config.ToolRegistries = []string{"github.com/obot-platform/tools"}


github.com/gptscript-ai/cmd's struct tag defaulting for []slice types doesn't work. Setting it here until I can push a fix to that package.

njhale · 2025-01-15T08:57:47Z

tools/package-tools.sh

+cd obot-tools
+cat <<EOF > .envrc.tools
+export GPTSCRIPT_SYSTEM_TOOLS_DIR=/obot-tools/
+export GPTSCRIPT_TOOL_REMAP="$(IFS=','; echo "${REGISTRY_REMAP[*]}")"
+export OBOT_SERVER_TOOL_REGISTRIES="${TOOL_REGISTRY_REPOS}"
+export OBOT_SERVER_VERSIONS="${OBOT_SERVER_VERSIONS}"
+export TOOLS_VENV_BIN=/obot-tools/venv/bin


not the prettiest solution -- and breaks outside the context of the container image since all the paths assume are absolute off of the root dir -- but I think it de-spaghettifies the construction of these settings a bit

Support consuming tools from more than one tool registry at a time. Replaces the `OBOT_SERVER_TOOL_REGISTRY` option with `OBOT_SERVER_TOOL_REGISTRIES`. The new option accepts a comma-delimited string of registry URIs. e.g. ```bash export OBOT_SERVER_TOOL_REGISTRIES='github.com/obot-platform/tools,github.com/obot-platform/enterprise-tools'` ``` When overriding a remote registry with a directory in the local filesystem, the `GPTSCRIPT_TOOL_REMAP` environment variable is used to provide tool reference remapping in GPTScript/Obot. e.g. Replacing `github.com/obot-platform/tools` with `../tools`: ```bash export GPTSCRIPT_TOOL_REMAP='github.com/obot-platform/tools=../tools' ``` e.g. Replacing `github.com/obot-platform/tools` with `../tools` and `github.com/obot-platform/enterprise-tools` with `../enterprise-tools`: ```bash export GPTSCRIPT_TOOL_REMAP='github.com/obot-platform/tools=../tools,github.com/obot-platform/enterprise-tools=../enterprise-tools' export OBOT_SERVER_TOOL_REGISTRIES='github.com/obot-platform/tools,github.com/obot-platform/enterprise-tools'` ``` Note: - `OBOT_SERVER_TOOL_REGISTRIES` should always reference remote tool registry repos, never local file paths - Only `GPTSCRIPT_TOOL_REMAP` needs to be set to swap the default tool registry (`github.com/obot-platform/tools`) with a local fork Signed-off-by: Nick Hale <[email protected]>

Add support for: - Pulling multiple tool registries into the image at build-time via the `TOOL_REGISTY_REPOS` build arg - Cloning private tool registry repos via the `GITHUB_TOKEN` Docker secret e.g. Building the "enterprise" Obot image ```bash export GITHUB_TOKEN=$(gh auth token) docker build --build-arg TOOL_REGISTRY_REPOS='github.com/obot-platform/enterprise-tools,github.com/obot-platform/tools' \ --secret id=GITHUB_TOKEN \ -t obot-enterprise:latest . ``` Signed-off-by: Nick Hale <[email protected]>

njhale changed the title ~~enhance/multiple tool registries~~ Support enterprise tool registry Jan 9, 2025

njhale commented Jan 9, 2025

View reviewed changes

.github/workflows/docker-build-and-push.yml Outdated Show resolved Hide resolved

njhale commented Jan 9, 2025

View reviewed changes

tools/package-tools.sh Outdated Show resolved Hide resolved

njhale force-pushed the enhance/multiple-tool-registries branch from dbd6582 to d9f6434 Compare January 9, 2025 22:32

njhale commented Jan 9, 2025

View reviewed changes

tools/package-tools.sh Outdated Show resolved Hide resolved

njhale force-pushed the enhance/multiple-tool-registries branch 5 times, most recently from 9682fd7 to 2c704cd Compare January 13, 2025 21:57

cjellick reviewed Jan 13, 2025

View reviewed changes

pkg/storage/server.go Show resolved Hide resolved

njhale force-pushed the enhance/multiple-tool-registries branch 5 times, most recently from 4c2258b to c928b8f Compare January 15, 2025 08:35

njhale requested review from ibuildthecloud, cjellick, drpebcak, thedadams, ivyjeong13, StrongMonkey, iwilltry42, ryanhopperlowe and g-linville January 15, 2025 08:39

njhale marked this pull request as ready for review January 15, 2025 08:39

njhale commented Jan 15, 2025

View reviewed changes

njhale force-pushed the enhance/multiple-tool-registries branch from c928b8f to addb333 Compare January 15, 2025 18:10

njhale mentioned this pull request Jan 15, 2025

chore: build and publish enterprise images #1268

Merged

njhale changed the title ~~Support enterprise tool registry~~ Support multiple tool registries Jan 15, 2025

njhale force-pushed the enhance/multiple-tool-registries branch from addb333 to 2868c13 Compare January 15, 2025 21:39

drpebcak approved these changes Jan 16, 2025

View reviewed changes

StrongMonkey approved these changes Jan 16, 2025

View reviewed changes

njhale added 2 commits January 16, 2025 19:27

njhale force-pushed the enhance/multiple-tool-registries branch from 2868c13 to 3590aed Compare January 17, 2025 00:47

njhale merged commit 42c02c3 into obot-platform:main Jan 17, 2025
2 checks passed

njhale deleted the enhance/multiple-tool-registries branch January 17, 2025 01:25

njhale mentioned this pull request Jan 17, 2025

feat: auth providers #1085

Open

njhale restored the enhance/multiple-tool-registries branch January 17, 2025 13:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support multiple tool registries #1193

Support multiple tool registries #1193

njhale commented Jan 9, 2025 •

edited

Loading

njhale commented Jan 15, 2025

njhale commented Jan 15, 2025

njhale Jan 15, 2025

njhale Jan 15, 2025

njhale Jan 15, 2025

njhale Jan 15, 2025

njhale Jan 15, 2025

		WorkspaceTool string `usage:"The tool reference for the workspace provider" default:"github.com/gptscript-ai/workspace-provider"`
		DatasetsTool string `usage:"The tool reference for the dataset provider" default:"github.com/gptscript-ai/datasets"`

Support multiple tool registries #1193

Support multiple tool registries #1193

Conversation

njhale commented Jan 9, 2025 • edited Loading

njhale commented Jan 15, 2025

njhale commented Jan 15, 2025

njhale Jan 15, 2025

Choose a reason for hiding this comment

njhale Jan 15, 2025

Choose a reason for hiding this comment

njhale Jan 15, 2025

Choose a reason for hiding this comment

njhale Jan 15, 2025

Choose a reason for hiding this comment

njhale Jan 15, 2025

Choose a reason for hiding this comment

njhale commented Jan 9, 2025 •

edited

Loading