Skip to content

Commit

Permalink
Refactor /codegen/build-clients.sh to support new apis structure,…
Browse files Browse the repository at this point in the history
… regenerate for `2024-10` (#76)

## Problem
The upstream `/apis` submodule has undergone some changes since we last
regenerated the client core. Our `/codegen/build-clients.sh` script no
longer handles the new module naming or structure properly.

Additionally, we want to regenerate off the upcoming API version so we
can pull in changes for things like Rerank.

This PR is **merging to a release candidate (RC) branch:
`release-candidate/2024-10`**. We will track upcoming major version
changes in this branch and use it for releasing dev builds.

## Solution
- Refactor `codegen/build-clients.sh` script to support the newer
structure for the upcoming API version, specifically `inference` is its
own module, and `db_control`/`db_data` as separate modules.
- Update imports and exports for `control` -> `db_control`, and
`control` -> `inference` for the `Embed()` method on `InferenceService`.

We made the decision to have `InferenceService` as its own module inside
`Client` because of the split in the API spec, so this was fairly easy
to refactor around here.

## Type of Change
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] This change requires a documentation update
- [X] Infrastructure change (CI configs, etc)
- [ ] Non-code change (docs, etc)
- [ ] None of the above: (explain here)

## Test Plan
The code changes under `/internal/gen/*` are all from running the new
`/codegen/build-clients.sh` script. I changed files in the submodule for
`apis` manually for generating, but it aligns with this PR:
pinecone-io/apis#138

Make sure CI build + tests pass for the new core stuff here.
  • Loading branch information
austin-denoble committed Oct 23, 2024
1 parent 7c4bc23 commit 6886f84
Show file tree
Hide file tree
Showing 13 changed files with 3,009 additions and 685 deletions.
4 changes: 1 addition & 3 deletions .github/workflows/ci.yaml
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
name: ci
on:
pull_request:
branches:
- main
pull_request: {}

jobs:
build:
Expand Down
2 changes: 1 addition & 1 deletion codegen/apis
Submodule apis updated from 062b11 to 3b7369
71 changes: 51 additions & 20 deletions codegen/build-clients.sh
Original file line number Diff line number Diff line change
@@ -1,14 +1,26 @@
#!/bin/bash

set -eux -o pipefail

version=$1 # e.g. 2024-07

# data_destination must align with the option go_package:
# https://github.com/pinecone-io/apis/blob/e9b47c76f649656002f4911946ca6c4c4a6f04fc/src/release/data/data.proto#L3
data_destination="internal/gen/data"
control_destination="internal/gen/control"
# modules
db_control_module="db_control"
db_data_module="db_data"
inference_module="inference"

# generated output destination paths
# db_data_destination must align with the option go_package in the proto file:
# https://github.com/pinecone-io/apis/blob/d1d005e75cc9fe9a5c486ef9218fe87b57765961/src/release/db/data/data.proto#L3
db_data_destination="internal/gen/data"
db_control_destination="internal/gen/${db_control_module}"
inference_destination="internal/gen/${inference_module}"

# version file
version_file="internal/gen/api_version.go"
# generated oas files
db_control_oas_file="${db_control_destination}/${db_control_module}_${version}.oas.go"
inference_oas_file="${inference_destination}/${inference_module}_${version}.oas.go"

set -eux -o pipefail

update_apis_repo() {
echo "Updating apis repo"
Expand All @@ -27,18 +39,35 @@ verify_spec_version() {
echo "Version is required"
exit 1
fi

verify_directory_exists "codegen/apis/_build/${version}"
}

verify_directory_exists() {
local directory=$1
if [ ! -d "$directory" ]; then
echo "Directory does not exist at $directory"
exit 1
fi
}

generate_oas_client() {
oas_file="codegen/apis/_build/${version}/control_${version}.oas.yaml"
local module=$1
local destination=$2

# source oas file for module and version
oas_file="codegen/apis/_build/${version}/${module}_${version}.oas.yaml"

oapi-codegen --package=control \
oapi-codegen --package=${module} \
--generate types,client \
"${oas_file}" > "${control_destination}/control_plane.oas.go"
"${oas_file}" > "${destination}"
}

generate_proto_client() {
proto_file="codegen/apis/_build/${version}/data_${version}.proto"
local module=$1

# source proto file for module and version
proto_file="codegen/apis/_build/${version}/${module}_${version}.proto"

protoc --experimental_allow_proto3_optional \
--proto_path=codegen/apis/vendor/protos \
Expand All @@ -63,19 +92,21 @@ EOL
update_apis_repo
verify_spec_version $version

# Generate control plane client code
rm -rf "${control_destination}"
mkdir -p "${control_destination}"

generate_oas_client
# Generate db_control oas client
rm -rf "${db_control_destination}"
mkdir -p "${db_control_destination}"
generate_oas_client $db_control_module $db_control_oas_file

# Generate data plane client code
rm -rf "${data_destination}"
mkdir -p "${data_destination}"
# Generate inference oas client
rm -rf "${inference_destination}"
mkdir -p "${inference_destination}"
generate_oas_client $inference_module $inference_oas_file

generate_proto_client
# Generate db_data proto client
rm -rf "${db_data_destination}"
mkdir -p "${db_data_destination}"
generate_proto_client $db_data_module

# Generate version file
rm -rf "${version_file}"

generate_version_file
2 changes: 1 addition & 1 deletion internal/gen/api_version.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

204 changes: 2 additions & 202 deletions internal/gen/control/control_plane.oas.go

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading

0 comments on commit 6886f84

Please sign in to comment.