From 971943b4063752bed5b1c6b77180305ad8c1ec83 Mon Sep 17 00:00:00 2001 From: Tim Burks Date: Fri, 18 Feb 2022 14:55:46 -0800 Subject: [PATCH] Replace "reference implementation" with "core implementation" in README. (#467) Also include results of running tools/pretty.sh to format all READMEs. --- README.md | 8 +- .../registry-lint-openapi-sample/README.md | 6 +- log/README.md | 4 +- tests/controller/README.md | 149 +++++++++++++----- 4 files changed, 123 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 578723409..5f38aa143 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,8 @@ [![Go Actions Status](https://github.com/apigee/registry/workflows/Go/badge.svg)](https://github.com/apigee/registry/actions) -# Registry API Reference Implementation +# Registry API Core Implementation -This repository contains a reference implementation of the Registry API. +This repository contains the core implementation of the Registry API. ## The Registry API @@ -27,8 +27,8 @@ tell us about your experience if you use it. ## This Implementation -This reference implementation is a [gRPC](https://grpc.io) service written in -Go. It can be run locally or deployed in a container using services including +This implementation is a [gRPC](https://grpc.io) service written in Go. It can +be run locally or deployed in a container using services including [Google Cloud Run](https://cloud.google.com/run). It stores data using a configurable relational interface layer that currently supports [PostgreSQL](https://www.postgresql.org/) and diff --git a/cmd/registry/plugins/registry-lint-openapi-sample/README.md b/cmd/registry/plugins/registry-lint-openapi-sample/README.md index 9a6242b15..0dc5cc7c6 100644 --- a/cmd/registry/plugins/registry-lint-openapi-sample/README.md +++ b/cmd/registry/plugins/registry-lint-openapi-sample/README.md @@ -1,4 +1,6 @@ The sample OpenAPI linter in this directory enforces the following rules: -`description-less-than-1000-chars`: Ensures that description fields in the OpenAPI spec are less than 1000 characters. -`description-contains-no-tags`: Ensures that description fields in the OpenAPI spec contain no tags (such as HTML tags). +`description-less-than-1000-chars`: Ensures that description fields in the +OpenAPI spec are less than 1000 characters. `description-contains-no-tags`: +Ensures that description fields in the OpenAPI spec contain no tags (such as +HTML tags). diff --git a/log/README.md b/log/README.md index 3dcc7fd5e..bc8e671d1 100644 --- a/log/README.md +++ b/log/README.md @@ -1,3 +1,5 @@ # log -This package provides logging for the Registry project. See https://pkg.go.dev/github.com/apigee/registry/log for package documentation and examples. \ No newline at end of file +This package provides logging for the Registry project. See +https://pkg.go.dev/github.com/apigee/registry/log for package documentation and +examples. diff --git a/tests/controller/README.md b/tests/controller/README.md index e5da6e484..93ddf7a85 100644 --- a/tests/controller/README.md +++ b/tests/controller/README.md @@ -1,11 +1,21 @@ ## Registry Controller -In simple terms, controller is a framework to keep the registry up-to-date. The idea for this framework is inspired from the [Kubernetes controller](https://kubernetes.io/docs/concepts/architecture/controller/) pattern. Controller can be used to enrich the API data stored in the registry, with additional analytical artifacts. +In simple terms, controller is a framework to keep the registry up-to-date. The +idea for this framework is inspired from the +[Kubernetes controller](https://kubernetes.io/docs/concepts/architecture/controller/) +pattern. Controller can be used to enrich the API data stored in the registry, +with additional analytical artifacts. ### Configuration details -A Controller will take in configuration (aka manifest) which describes the desired state of the registry. The desired state in this context is essentially dependency relations between different entities in the database. The controller’s job is to keep these dependency relations valid and up-to-date. It will take actions to eventually get the registry to its desired state. + +A Controller will take in configuration (aka manifest) which describes the +desired state of the registry. The desired state in this context is essentially +dependency relations between different entities in the database. The +controller’s job is to keep these dependency relations valid and up-to-date. It +will take actions to eventually get the registry to its desired state. Example manifest: + ``` id: "example-manifest" generated_resources: @@ -15,28 +25,46 @@ generated_resources: filter: "mime_type.contains('openapi')" action: "registry compute lint $resource.spec --linter spectral" ``` -The above manifest defines that for every spec (`pattern: $resource.spec`) of type openapi (`filter: "mime_type.contains('openapi')"`) in the registry, there should exist a corresponding lint-spectral artifact (`pattern: apis/-/versions/-/specs/-/artifacts/lint-spectral`). If the target artifact is missing or out-of-date, then it should be re-generated using the specified action (`action: "registry compute lint $resource.spec --linter spectral"`). -The controller will compare the current and the supplied desired state of the registry. For resources which are non-existing or outdated, the controller will execute the action supplied in the manifest. +The above manifest defines that for every spec (`pattern: $resource.spec`) of +type openapi (`filter: "mime_type.contains('openapi')"`) in the registry, there +should exist a corresponding lint-spectral artifact +(`pattern: apis/-/versions/-/specs/-/artifacts/lint-spectral`). If the target +artifact is missing or out-of-date, then it should be re-generated using the +specified action +(`action: "registry compute lint $resource.spec --linter spectral"`). + +The controller will compare the current and the supplied desired state of the +registry. For resources which are non-existing or outdated, the controller will +execute the action supplied in the manifest. ### Usage -With this basic definition in mind, you can supply different configurations to the controller to generate various artifacts in the registry. +With this basic definition in mind, you can supply different configurations to +the controller to generate various artifacts in the registry. #### Setup the registry server -Make sure you have a registry server running. We will deploy the server to GKE for the purpose of this walkthrough. -- Deploy the registry server to GKE. The following two make targets will build and deploy the server: +Make sure you have a registry server running. We will deploy the server to GKE +for the purpose of this walkthrough. + +- Deploy the registry server to GKE. The following two make targets will build + and deploy the server: + ```shell # Build make build - + # Deploy make deploy-gke - ``` - Follow these [instructions](https://github.com/apigee/registry/blob/main/README.md#running-the-registry-api-server-on-gke) for more details. + ``` + + Follow these + [instructions](https://github.com/apigee/registry/blob/main/README.md#running-the-registry-api-server-on-gke) + for more details. - Populate some resources in the registry: + ```shell # Setup auth source auth/GKE.sh @@ -49,14 +77,17 @@ Make sure you have a registry server running. We will deploy the server to GKE f # Create 10 versions of the petstore API ./tests/controller/create_apis.sh 10 - ``` + ``` #### Upload the manifest + ```shell registry upload manifest tests/controller/testdata/manifest.yaml --project-id=demo ``` + The contents of the manifest that we are uploading is as follows: + ```yaml id: "test-manifest" generated_resources: @@ -81,47 +112,76 @@ generated_resources: ``` #### Run the controller + There are three ways to run the controller: ##### Dry-run mode: -In this mode, the controller will generate the actions, which it calculates, are needed to bring registry up-to-date. However, it will not execute any of this actions. This is a way to try and see what the controller is doing without making any modifications to the registry. The controller can be invoked by using the `registry resolve` command: + +In this mode, the controller will generate the actions, which it calculates, +are needed to bring registry up-to-date. However, it will not execute any of +this actions. This is a way to try and see what the controller is doing without +making any modifications to the registry. The controller can be invoked by +using the `registry resolve` command: + ```shell registry resolve projects/demo/locations/global/artifacts/test-manifest --dry-run ``` + The above command should generate 30 actions: -* 10 `registry compute lint ...` -* 10 `registry compute vocabulary ...` -* 10 `registry compute complexity ...` -* `registry compute lintstats ...` is omitted because the dependencies for that are non-existent at this point. +- 10 `registry compute lint ...` +- 10 `registry compute vocabulary ...` +- 10 `registry compute complexity ...` +- `registry compute lintstats ...` is omitted because the dependencies for that + are non-existent at this point. ##### Standalone mode: -In this mode, we let the controller go through one iteration of the comparison of current and desired states. For each iteration, the controller generates and executes a set of actions which takes the registry closer to it's desired state. + +In this mode, we let the controller go through one iteration of the comparison +of current and desired states. For each iteration, the controller generates and +executes a set of actions which takes the registry closer to it's desired +state. ###### Iteration 1: + ```shell registry resolve projects/demo/locations/global/artifacts/test-manifest ``` -This will generate and execute the same 30 actions mentioned above. You can see the artifacts generated by the controller by doing a list call + +This will generate and execute the same 30 actions mentioned above. You can see +the artifacts generated by the controller by doing a list call `registry list projects/demo/locations/global/apis/-/versions/-/specs/-/artifacts/-` ###### Iteration 2: -Note that we haven't generated the additional 10 lintstats-spectral artifacts yet. + +Note that we haven't generated the additional 10 lintstats-spectral artifacts +yet. + ```shell registry resolve projects/demo/locations/global/artifacts/test-manifest ``` -After the first iteration, the dependencies (`$resource.spec/artifacts/lint-spectral`) are generated, and hence in this iteration, the controller will generate and execute 10 actions. -* 10 `registry compute lintstats ...` -A quick check with a list call can verify things -`registry list projects/demo/locations/global/apis/-/versions/-/specs/-/artifacts/-` -To summarize, in the standalone mode, the controller can bring the registry up-to-date in two iterations. +After the first iteration, the dependencies +(`$resource.spec/artifacts/lint-spectral`) are generated, and hence in this +iteration, the controller will generate and execute 10 actions. + +- 10 `registry compute lintstats ...` A quick check with a list call can verify + things + `registry list projects/demo/locations/global/apis/-/versions/-/specs/-/artifacts/-` + +To summarize, in the standalone mode, the controller can bring the registry +up-to-date in two iterations. ##### Continuous mode: -In this mode, the controller is running continuously, making sure that it is always checking the state of the registry in each passing iteration. This can be achieved through a GKE cron job. -###### Step 1: Building the docker image: +In this mode, the controller is running continuously, making sure that it is +always checking the state of the registry in each passing iteration. This can +be achieved through a GKE cron job. + +###### Step 1: Building the docker image: + Build the base image which includes the registry tool: + ```shell # Build docker build -f containers/registry-tools/Dockerfile . -t gcr.io/$REGISTRY_PROJECT_IDENTIFIER/registry-tools @@ -130,7 +190,9 @@ docker build -f containers/registry-tools/Dockerfile . -t gcr.io/$REGISTRY_PROJE docker push gcr.io/$REGISTRY_PROJECT_IDENTIFIER/registry-tools:latest ``` -Build the linter image from the base image which includes the binaries necessary for the `test-manifest` to work. +Build the linter image from the base image which includes the binaries +necessary for the `test-manifest` to work. + ``` # Build envsubst < containers/registry-tools/registry-linters/Dockerfile | docker build -t gcr.io/$REGISTRY_PROJECT_IDENTIFIER/registry-linters -f - . @@ -140,7 +202,10 @@ docker push gcr.io/$REGISTRY_PROJECT_IDENTIFIER/registry-linters:latest ``` ###### Step 2: Deploying the job -Deploy the controller job which will use the `registry-linters` image and run the `registry resolve` command, every 5 mins + +Deploy the controller job which will use the `registry-linters` image and run +the `registry resolve` command, every 5 mins + ```shell export REGISTRY_MANIFEST_ID=projects/demo/locations/global/artifacts/test-manifest @@ -148,19 +213,28 @@ make deploy-controller-job ``` ###### Step 3: Verification -* Once the job is deployed, you should see that in every iteration it is reading the manifest and calculating the actions. -* Since, we already have the artifacts generated from our previous commands, we need to clean up the previously generated artifacts. -`registry delete projects/demo/locations/global/apis/-/versions/-/specs/-/artifacts/-`. -Once the artifacts are cleaned up, we can see the controller in action. -* Use list command to check what artifacts are generated by the controller, the behavior should be the same as the one described in the standalone case. -`registry list projects/demo/locations/global/apis/-/versions/-/specs/-/artifacts/-` -* You now have a controller continuously running in your GKE project which will be responsible for generating and maintaining the artifacts defined in the manifest. +- Once the job is deployed, you should see that in every iteration it is + reading the manifest and calculating the actions. +- Since, we already have the artifacts generated from our previous commands, we + need to clean up the previously generated artifacts. + `registry delete projects/demo/locations/global/apis/-/versions/-/specs/-/artifacts/-`. + Once the artifacts are cleaned up, we can see the controller in action. +- Use list command to check what artifacts are generated by the controller, the + behavior should be the same as the one described in the standalone case. + `registry list projects/demo/locations/global/apis/-/versions/-/specs/-/artifacts/-` +- You now have a controller continuously running in your GKE project which will + be responsible for generating and maintaining the artifacts defined in the + manifest. #### Desired state definitions: ##### Recommended manifest: -To get started with the controller framework, we recommend using the following manifest. It defines four artifacts and all of them can be generated using the `registry tool`. This is the same manifest used in the walkthrough above. + +To get started with the controller framework, we recommend using the following +manifest. It defines four artifacts and all of them can be generated using the +`registry tool`. This is the same manifest used in the walkthrough above. + ```yaml id: "test-manifest" generated_resources: @@ -184,7 +258,9 @@ generated_resources: ``` ##### Projects with protobuf spec definitions: + If your project contains protobuf definitions, use the api-linter. + ```yaml id: "test-manifest" generated_resources: @@ -211,4 +287,3 @@ generated_resources: - pattern: "$resource.spec" action: "registry compute complexity $resource.spec" ``` -