Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

implement gRPC rest gateway #39

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,14 @@ PATH := $(PROJECT_BIN):$(PATH)
model-registry: build

internal/ml_metadata/proto/%.pb.go: api/grpc/ml_metadata/proto/%.proto
protoc -I./api/grpc --go_out=./internal --go_opt=paths=source_relative \
--go-grpc_out=./internal --go-grpc_opt=paths=source_relative $<
protoc -I./api/grpc \
--go_out=./internal --go_opt=paths=source_relative \
--go-grpc_out=./internal --go-grpc_opt=paths=source_relative \
--grpc-gateway_out=./internal --grpc-gateway_opt logtostderr=true \
--grpc-gateway_opt paths=source_relative --grpc-gateway_opt generate_unbound_methods=true \
--openapiv2_out ./api/gen/openapiv2 --openapiv2_opt logtostderr=true --openapiv2_opt json_names_for_fields=false \
--openapiv2_opt generate_unbound_methods=true --openapiv2_opt output_format=yaml \
--openapiv2_opt preserve_rpc_order=true $<

.PHONY: gen/grpc
gen/grpc: internal/ml_metadata/proto/metadata_store.pb.go internal/ml_metadata/proto/metadata_store_service.pb.go
Expand All @@ -26,7 +32,7 @@ vet:

.PHONY: clean
clean:
rm -Rf ./model-registry internal/ml_metadata/proto/*.go internal/model/graph/models_gen.go
rm -Rf ./model-registry internal/ml_metadata/proto/*.go internal/model/graph/models_gen.go api/gen/openapiv2/ml_metadata/proto/*.yaml

.PHONY: deps
deps:
Expand All @@ -35,6 +41,8 @@ deps:
GOBIN=$(PROJECT_BIN) go install github.com/searKing/golang/tools/[email protected]
GOBIN=$(PROJECT_BIN) go install google.golang.org/protobuf/cmd/[email protected]
GOBIN=$(PROJECT_BIN) go install google.golang.org/grpc/cmd/[email protected]
GOBIN=$(PROJECT_BIN) go install github.com/grpc-ecosystem/grpc-gateway/v2/[email protected]
GOBIN=$(PROJECT_BIN) go install github.com/grpc-ecosystem/grpc-gateway/v2/[email protected]

.PHONY: vendor
vendor:
Expand Down Expand Up @@ -67,6 +75,10 @@ run/server: gen metadata.sqlite.db
run/client: gen
python test/python/test_mlmetadata.py

.PHONY: run/gateway
run/gateway:
go run main.go gateway --logtostderr=true

.PHONY: serve
serve: build
./model-registry serve --logtostderr=true
Expand Down
144 changes: 144 additions & 0 deletions api/gen/openapiv2/ml_metadata/proto/metadata_store.swagger.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
swagger: "2.0"
info:
title: ml_metadata/proto/metadata_store.proto
version: version not set
consumes:
- application/json
produces:
- application/json
paths: {}
definitions:
protobufAny:
type: object
properties:
'@type':
type: string
description: |-
A URL/resource name that uniquely identifies the type of the serialized
protocol buffer message. This string must contain at least
one "/" character. The last segment of the URL's path must represent
the fully qualified name of the type (as in
`path/google.protobuf.Duration`). The name should be in a canonical form
(e.g., leading "." is not accepted).

In practice, teams usually precompile into the binary all types that they
expect it to use in the context of Any. However, for URLs which use the
scheme `http`, `https`, or no scheme, one can optionally set up a type
server that maps type URLs to message definitions as follows:

* If no scheme is provided, `https` is assumed.
* An HTTP GET on the URL must yield a [google.protobuf.Type][]
value in binary format, or produce an error.
* Applications are allowed to cache lookup results based on the
URL, or have them precompiled into a binary to avoid any
lookup. Therefore, binary compatibility needs to be preserved
on changes to types. (Use versioned type names to manage
breaking changes.)

Note: this functionality is not currently available in the official
protobuf release, and it is not used for type URLs beginning with
type.googleapis.com. As of May 2023, there are no widely used type server
implementations and no plans to implement one.

Schemes other than `http`, `https` (or the empty scheme) might be
used with implementation specific semantics.
additionalProperties: {}
description: |-
`Any` contains an arbitrary serialized protocol buffer message along with a
URL that describes the type of the serialized message.

Protobuf library provides support to pack/unpack Any values in the form
of utility functions or additional generated methods of the Any type.

Example 1: Pack and unpack a message in C++.

Foo foo = ...;
Any any;
any.PackFrom(foo);
...
if (any.UnpackTo(&foo)) {
...
}

Example 2: Pack and unpack a message in Java.

Foo foo = ...;
Any any = Any.pack(foo);
...
if (any.is(Foo.class)) {
foo = any.unpack(Foo.class);
}
// or ...
if (any.isSameTypeAs(Foo.getDefaultInstance())) {
foo = any.unpack(Foo.getDefaultInstance());
}

Example 3: Pack and unpack a message in Python.

foo = Foo(...)
any = Any()
any.Pack(foo)
...
if any.Is(Foo.DESCRIPTOR):
any.Unpack(foo)
...

Example 4: Pack and unpack a message in Go

foo := &pb.Foo{...}
any, err := anypb.New(foo)
if err != nil {
...
}
...
foo := &pb.Foo{}
if err := any.UnmarshalTo(foo); err != nil {
...
}

The pack methods provided by protobuf library will by default use
'type.googleapis.com/full.type.name' as the type URL and the unpack
methods only use the fully qualified type name after the last '/'
in the type URL, for example "foo.bar.com/x/y.z" will yield type
name "y.z".

JSON
====
The JSON representation of an `Any` value uses the regular
representation of the deserialized, embedded message, with an
additional field `@type` which contains the type URL. Example:

package google.profile;
message Person {
string first_name = 1;
string last_name = 2;
}

{
"@type": "type.googleapis.com/google.profile.Person",
"firstName": <string>,
"lastName": <string>
}

If the embedded message type is well-known and has a custom JSON
representation, that representation will be embedded adding a field
`value` which holds the custom JSON in addition to the `@type`
field. Example (for message [google.protobuf.Duration][]):

{
"@type": "type.googleapis.com/google.protobuf.Duration",
"value": "1.212s"
}
rpcStatus:
type: object
properties:
code:
type: integer
format: int32
message:
type: string
details:
type: array
items:
type: object
$ref: '#/definitions/protobufAny'
Loading